2024-01-11T09:55:38.317 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-11T09:55:38.318 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429 --owner scheduled_matan@teuthology --description crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} --name matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi -- /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/orig.config.yaml 2024-01-11T09:55:38.362 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T09:55:38.446 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429 branch: wip-matanb-crimson-multicore-msgr description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} email: null first_in_suite: false job_id: '7513429' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-multicore-msgr ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd failsafe full ratio: 0.95 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: bluestore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 4e3957a13c3b8938fe41053b13fb80161e1916b7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 4e3957a13c3b8938fe41053b13fb80161e1916b7 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: f14f60afd567d61490bfe92caba04c0ffb5b09d2 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 109 sha1: 4e3957a13c3b8938fe41053b13fb80161e1916b7 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: f14f60afd567d61490bfe92caba04c0ffb5b09d2 targets: smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9FCjzta3j/kTl2AOjmnxEItzktKfjKINZVSKfOMgwX2Ba7WsHwXv+m5Q1BXECZwcz6kkuR8ASgaWKtlP9UxeI= smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEp1dt+7AFCwJg78Y81xcft8Bn5sEh6vOHvj2vbxh5jm9Qm1g1pyyMHmf1Suc/91Z1i3t0pR8S2AcXCmiT4zS4= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson - thrashosds: ceph_objectstore_tool: false chance_down: 0 chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-11_09:46:59 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-11T09:55:38.447 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa; will attempt to use it 2024-01-11T09:55:38.447 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks 2024-01-11T09:55:38.448 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-11T09:55:38.449 INFO:teuthology.task.internal:Checking packages... 2024-01-11T09:55:38.479 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '4e3957a13c3b8938fe41053b13fb80161e1916b7' 2024-01-11T09:55:38.479 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-11T09:55:38.479 INFO:teuthology.packaging:ref: None 2024-01-11T09:55:38.480 INFO:teuthology.packaging:tag: None 2024-01-11T09:55:38.480 INFO:teuthology.packaging:branch: wip-matanb-crimson-multicore-msgr 2024-01-11T09:55:38.480 INFO:teuthology.packaging:sha1: 4e3957a13c3b8938fe41053b13fb80161e1916b7 2024-01-11T09:55:38.480 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-multicore-msgr 2024-01-11T09:55:38.731 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-635.g4e3957a1 2024-01-11T09:55:38.733 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-11T09:55:38.739 INFO:teuthology.task.internal:no buildpackages task found 2024-01-11T09:55:38.740 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-11T09:55:38.765 INFO:teuthology.task.internal:Saving configuration 2024-01-11T09:55:38.781 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-11T09:55:38.787 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T09:55:38.808 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 09:50:36.206849', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9FCjzta3j/kTl2AOjmnxEItzktKfjKINZVSKfOMgwX2Ba7WsHwXv+m5Q1BXECZwcz6kkuR8ASgaWKtlP9UxeI='} 2024-01-11T09:55:38.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 09:50:36.208506', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEp1dt+7AFCwJg78Y81xcft8Bn5sEh6vOHvj2vbxh5jm9Qm1g1pyyMHmf1Suc/91Z1i3t0pR8S2AcXCmiT4zS4='} 2024-01-11T09:55:38.831 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-11T09:55:38.837 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2024-01-11T09:55:38.837 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-01-11T09:55:38.837 INFO:teuthology.run_tasks:Running task console_log... 2024-01-11T09:55:38.907 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7a003c9ee0>, signals=[15]) 2024-01-11T09:55:38.907 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-11T09:55:38.913 INFO:teuthology.task.internal:Opening connections... 2024-01-11T09:55:38.913 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T09:55:38.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:55:39.022 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T09:55:39.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:55:39.116 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-11T09:55:39.122 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-01-11T09:55:39.140 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-01-11T09:55:39.140 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-01-11T09:55:39.197 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="CentOS Stream" 2024-01-11T09:55:39.197 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="9" 2024-01-11T09:55:39.197 INFO:teuthology.orchestra.run.smithi040.stdout:ID="centos" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE="rhel fedora" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="9" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:PLATFORM_ID="platform:el9" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:ANSI_COLOR="0;31" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:LOGO="fedora-logo-icon" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://centos.org/" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-11T09:55:39.198 INFO:teuthology.orchestra.run.smithi040.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-11T09:55:39.199 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-01-11T09:55:39.220 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-01-11T09:55:39.238 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-01-11T09:55:39.238 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-01-11T09:55:39.294 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-11T09:55:39.295 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-11T09:55:39.295 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-11T09:55:39.295 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-01-11T09:55:39.312 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-11T09:55:39.321 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-11T09:55:39.327 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-11T09:55:39.327 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-01-11T09:55:39.330 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-01-11T09:55:39.346 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-11T09:55:39.352 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-11T09:55:39.352 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-01-11T09:55:39.387 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-01-11T09:55:39.510 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-11T09:55:39.558 INFO:teuthology.run_tasks:Running task kernel... 2024-01-11T09:55:39.573 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-11T09:55:39.573 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-11T09:55:39.573 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T09:55:39.574 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-11T09:55:39.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:55:39.590 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-11T09:55:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:55:39.591 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-01-11T09:55:39.647 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-01-11T09:55:39.647 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-01-11T09:55:39.647 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-01-11T09:55:39.648 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-361.el9.x86_64 2024-01-11T09:55:39.649 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.14.0-361.el9.x86_64 2024-01-11T09:55:39.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum install -y kernel 2024-01-11T09:55:41.885 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 13 MB/s | 7.8 MB 00:00 2024-01-11T09:55:42.046 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 11 MB/s | 7.8 MB 00:00 2024-01-11T09:55:43.518 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 8.1 MB/s | 2.5 MB 00:00 2024-01-11T09:55:43.679 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.5 MB 00:00 2024-01-11T09:55:45.017 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 7.9 MB 00:01 2024-01-11T09:55:45.714 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 7.9 MB 00:01 2024-01-11T09:55:46.943 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 24 MB/s | 18 MB 00:00 2024-01-11T09:55:48.271 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 12 MB/s | 18 MB 00:01 2024-01-11T09:55:51.794 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 5.9 MB 00:01 2024-01-11T09:55:52.763 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 5.9 MB 00:01 2024-01-11T09:55:53.145 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 15 kB 00:00 2024-01-11T09:55:54.155 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2024-01-11T09:55:54.180 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 15 kB 00:00 2024-01-11T09:55:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 20 MB 00:02 2024-01-11T09:55:59.205 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-01-11T09:56:01.471 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T09:56:01.471 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T09:56:01.471 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T09:56:01.471 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.471 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T09:56:01.471 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.472 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.473 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T09:56:01.474 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.475 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.476 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.477 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.478 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.479 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.480 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T09:56:01.739 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-01-11T09:56:01.878 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-11T09:56:01.878 INFO:teuthology.orchestra.run.smithi040.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-11T09:56:01.982 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout: kernel x86_64 5.14.0-404.el9 baseos 5.9 M 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core x86_64 5.14.0-404.el9 baseos 21 M 2024-01-11T09:56:01.989 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules x86_64 5.14.0-404.el9 baseos 39 M 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core x86_64 5.14.0-404.el9 baseos 33 M 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout:Install 4 Packages 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 99 M 2024-01-11T09:56:01.990 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 122 M 2024-01-11T09:56:01.991 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T09:56:03.907 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T09:56:03.907 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T09:56:03.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T09:56:03.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T09:56:03.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T09:56:03.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T09:56:03.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T09:56:03.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T09:56:03.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.914 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.915 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.916 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.917 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T09:56:03.917 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.917 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.917 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T09:56:03.917 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T09:56:04.330 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-11T09:56:04.330 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-11T09:56:04.397 INFO:teuthology.orchestra.run.smithi040.stdout:(1/4): kernel-5.14.0-404.el9.x86_64.rpm 2.5 MB/s | 5.9 MB 00:02 2024-01-11T09:56:04.434 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T09:56:04.442 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-404.el9 baseos 5.9 M 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-404.el9 baseos 21 M 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-404.el9 baseos 39 M 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-404.el9 baseos 33 M 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-01-11T09:56:04.443 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:56:04.444 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-01-11T09:56:04.444 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 122 M 2024-01-11T09:56:04.444 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T09:56:07.247 INFO:teuthology.orchestra.run.smithi040.stdout:(2/4): kernel-core-5.14.0-404.el9.x86_64.rpm 4.0 MB/s | 21 MB 00:05 2024-01-11T09:56:08.907 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-404.el9.x86_64.rpm 1.3 MB/s | 5.9 MB 00:04 2024-01-11T09:56:09.431 INFO:teuthology.orchestra.run.smithi040.stdout:(3/4): kernel-modules-5.14.0-404.el9.x86_64.rpm 5.3 MB/s | 39 MB 00:07 2024-01-11T09:56:13.173 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-404.el9.x86_64.rpm 2.4 MB/s | 21 MB 00:08 2024-01-11T09:56:15.831 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-5.14.0-404.el9.x86_64.rpm 3.4 MB/s | 39 MB 00:11 2024-01-11T09:56:16.157 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-core-5.14.0-404.el9.x86_6 4.6 MB/s | 33 MB 00:07 2024-01-11T09:56:16.157 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T09:56:16.157 INFO:teuthology.orchestra.run.smithi134.stdout:Total 8.4 MB/s | 99 MB 00:11 2024-01-11T09:56:17.029 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T09:56:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T09:56:17.403 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T09:56:18.742 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T09:56:18.743 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T09:56:21.645 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T09:56:22.317 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-404.el9.x86_64 1/4 2024-01-11T09:56:22.448 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-404.el9.x86_64 2/4 2024-01-11T09:56:25.208 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-404.el9.x86_64 2/4 2024-01-11T09:56:25.385 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-404.el9.x86_64 3/4 2024-01-11T09:56:30.104 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-404.el9.x86_64 3/4 2024-01-11T09:56:30.294 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:56:34.304 INFO:teuthology.orchestra.run.smithi040.stdout:(4/4): kernel-modules-core-5.14.0-404.el9.x86_6 1.1 MB/s | 33 MB 00:29 2024-01-11T09:56:34.304 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T09:56:34.304 INFO:teuthology.orchestra.run.smithi040.stdout:Total 3.1 MB/s | 99 MB 00:32 2024-01-11T09:56:35.004 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:56:35.222 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T09:56:35.665 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T09:56:35.665 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T09:56:37.032 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T09:56:37.032 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T09:56:39.843 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T09:56:40.524 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-core-5.14.0-404.el9.x86_64 1/4 2024-01-11T09:56:40.654 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-core-5.14.0-404.el9.x86_64 2/4 2024-01-11T09:56:43.568 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-404.el9.x86_64 2/4 2024-01-11T09:56:43.756 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-modules-5.14.0-404.el9.x86_64 3/4 2024-01-11T09:56:48.817 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-404.el9.x86_64 3/4 2024-01-11T09:56:48.926 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : kernel-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:56:54.016 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-core-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:04.012 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-11T09:57:04.012 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:57:04.097 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:04.537 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:04.537 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-404.el9.x86_64 1/4 2024-01-11T09:57:04.537 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-404.el9.x86_64 2/4 2024-01-11T09:57:04.537 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-404.el9.x86_64 3/4 2024-01-11T09:57:05.020 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:05.020 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-404.el9.x86_64 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-404.el9.x86_64 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-404.el9.x86_64 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-404.el9.x86_64 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:57:05.021 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T09:57:05.530 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T09:57:05.675 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-404.el9.x86_64 2024-01-11T09:57:05.675 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-404.el9.x86_64 2024-01-11T09:57:05.675 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-404.el9.x86_64 2024-01-11T09:57:05.675 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-11T09:57:05.675 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-11T09:57:05.675 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-01-11T09:57:05.676 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-01-11T09:57:05.676 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T09:57:05.796 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-404.el9.x86_64 2024-01-11T09:57:05.796 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-404.el9.x86_64 2024-01-11T09:57:05.797 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-01-11T09:57:05.830 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-01-11T09:57:05.830 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-01-11T09:57:05.830 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-01-11T09:57:05.831 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T09:57:05.832 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-01-11T09:57:05.833 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-11T09:57:05.833 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-404.el9.x86_64 2024-01-11T09:57:05.833 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-11T09:57:06.415 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-01-11T09:57:07.782 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-11T09:57:07.802 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-01-11T09:57:07.804 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-01-11T09:57:07.823 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.tU84Enu87Z 2024-01-11T09:57:07.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tU84Enu87Z 2024-01-11T09:57:07.894 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.tU84Enu87Z 2024-01-11T09:57:08.039 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.tU84Enu87Z is 6KB 2024-01-11T09:57:08.061 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.tU84Enu87Z 2024-01-11T09:57:08.077 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-01-11T09:57:08.172 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-11T09:57:08.172 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-11T09:57:08.172 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-11T09:57:08.172 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-404.el9.x86_64.conf 2024-01-11T09:57:08.174 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-404.el9.x86_64 2024-01-11T09:57:08.330 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-01-11T09:57:24.700 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-core-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:24.700 INFO:teuthology.orchestra.run.smithi040.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-11T09:57:24.700 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:57:24.786 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-modules-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:25.287 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: kernel-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:25.287 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-5.14.0-404.el9.x86_64 1/4 2024-01-11T09:57:25.287 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-core-5.14.0-404.el9.x86_64 2/4 2024-01-11T09:57:25.287 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-5.14.0-404.el9.x86_64 3/4 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : kernel-modules-core-5.14.0-404.el9.x86_64 4/4 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-5.14.0-404.el9.x86_64 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-core-5.14.0-404.el9.x86_64 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-5.14.0-404.el9.x86_64 2024-01-11T09:57:25.749 INFO:teuthology.orchestra.run.smithi040.stdout: kernel-modules-core-5.14.0-404.el9.x86_64 2024-01-11T09:57:25.750 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:57:25.750 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T09:57:26.239 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T09:57:26.389 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-404.el9.x86_64 2024-01-11T09:57:26.390 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-404.el9.x86_64 2024-01-11T09:57:26.390 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-404.el9.x86_64 2024-01-11T09:57:26.390 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-11T09:57:26.390 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-11T09:57:26.390 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-11T09:57:26.390 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-01-11T09:57:26.390 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q kernel | sort -rV | head -n 1 2024-01-11T09:57:26.515 INFO:teuthology.orchestra.run.smithi040.stdout:kernel-5.14.0-404.el9.x86_64 2024-01-11T09:57:26.516 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-404.el9.x86_64 2024-01-11T09:57:26.516 DEBUG:teuthology.orchestra.run.smithi040:> sudo rpm -qi grub2-tools 2024-01-11T09:57:26.553 INFO:teuthology.orchestra.run.smithi040.stdout:Name : grub2-tools 2024-01-11T09:57:26.553 INFO:teuthology.orchestra.run.smithi040.stdout:Epoch : 1 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Version : 2.06 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Release : 68.el9 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: x86_64 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Group : Unspecified 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Size : 8264254 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:License : GPLv3+ 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Packager : builder@centos.org 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Vendor : CentOS 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-11T09:57:26.554 INFO:teuthology.orchestra.run.smithi040.stdout:Summary : Support tools for GRUB. 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:Description : 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:hardware devices. 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T09:57:26.555 INFO:teuthology.orchestra.run.smithi040.stdout:This subpackage provides tools for support of all platforms. 2024-01-11T09:57:26.556 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-11T09:57:26.556 INFO:teuthology.task.kernel:Updating grub on smithi040 to boot 5.14.0-404.el9.x86_64 2024-01-11T09:57:26.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-11T09:57:27.203 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-01-11T09:57:28.632 INFO:teuthology.orchestra.run.smithi040.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-11T09:57:28.667 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-01-11T09:57:28.670 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-01-11T09:57:28.727 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.Ebfeg2RRet 2024-01-11T09:57:28.728 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Ebfeg2RRet 2024-01-11T09:57:28.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.Ebfeg2RRet 2024-01-11T09:57:28.960 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.Ebfeg2RRet is 6KB 2024-01-11T09:57:29.009 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.Ebfeg2RRet 2024-01-11T09:57:29.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo /bin/ls /boot/loader/entries || true 2024-01-11T09:57:29.092 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-11T09:57:29.093 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-11T09:57:29.093 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-11T09:57:29.093 INFO:teuthology.orchestra.run.smithi040.stdout:15949e3c88704300882482eddfd7baae-5.14.0-404.el9.x86_64.conf 2024-01-11T09:57:29.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-404.el9.x86_64 2024-01-11T09:57:29.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-01-11T09:57:38.343 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-01-11T09:57:38.344 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-11T09:57:38.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:57:56.672 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-01-11T09:57:59.260 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-11T09:57:59.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-01-11T09:57:59.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:58:05.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-11T09:58:05.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:58:08.732 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-01-11T09:58:17.660 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-01-11T09:58:20.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-11T09:58:20.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:58:26.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-01-11T09:58:26.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:58:52.625 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-11T09:58:53.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-11T09:58:53.050 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-404.el9.x86_64"... 2024-01-11T09:58:53.050 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-01-11T09:58:53.064 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-404.el9.x86_64 2024-01-11T09:58:53.065 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-404.el9.x86_64 vs 5.14.0-404.el9.x86_64 2024-01-11T09:58:53.065 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T09:58:53.065 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-11T09:58:54.066 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-01-11T09:58:54.066 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-11T09:58:54.131 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-01-11T09:58:54.184 DEBUG:teuthology.parallel:result is None 2024-01-11T09:59:26.670 DEBUG:teuthology.orchestra.remote:timed out 2024-01-11T09:59:38.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-01-11T09:59:38.673 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T09:59:38.972 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-01-11T09:59:39.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-01-11T09:59:39.332 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-404.el9.x86_64"... 2024-01-11T09:59:39.333 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-01-11T09:59:39.348 INFO:teuthology.orchestra.run.smithi040.stdout:5.14.0-404.el9.x86_64 2024-01-11T09:59:39.349 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-404.el9.x86_64 vs 5.14.0-404.el9.x86_64 2024-01-11T09:59:39.349 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-11T09:59:39.349 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-11T09:59:40.350 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-11T09:59:40.350 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-11T09:59:40.422 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-01-11T09:59:40.467 DEBUG:teuthology.parallel:result is None 2024-01-11T09:59:40.468 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-11T09:59:40.475 INFO:teuthology.task.internal:Creating test directory... 2024-01-11T09:59:40.475 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T09:59:40.477 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-11T09:59:40.495 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-11T09:59:40.511 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-11T09:59:40.520 INFO:teuthology.task.internal:Creating archive directory... 2024-01-11T09:59:40.521 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T09:59:40.535 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-11T09:59:40.616 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-11T09:59:40.624 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-11T09:59:40.625 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T09:59:40.629 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-11T09:59:40.663 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T09:59:40.666 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T09:59:40.676 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T09:59:40.680 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-11T09:59:40.682 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-11T09:59:40.689 INFO:teuthology.task.internal:Configuring sudo... 2024-01-11T09:59:40.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T09:59:40.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-11T09:59:40.757 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-11T09:59:40.766 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-11T09:59:40.766 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T09:59:40.791 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-11T09:59:40.812 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T09:59:40.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T09:59:40.945 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T09:59:41.029 DEBUG:teuthology.orchestra.run.smithi040:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T09:59:41.073 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T09:59:41.073 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T09:59:41.141 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T09:59:41.192 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-11T09:59:41.240 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T09:59:41.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-11T09:59:41.370 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T09:59:41.370 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-11T09:59:41.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-01-11T09:59:41.442 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-01-11T09:59:41.487 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T09:59:41.530 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T09:59:42.019 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-11T09:59:42.029 INFO:teuthology.task.internal:Starting timer... 2024-01-11T09:59:42.029 INFO:teuthology.run_tasks:Running task pcp... 2024-01-11T09:59:42.043 INFO:teuthology.run_tasks:Running task selinux... 2024-01-11T09:59:42.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo service auditd rotate 2024-01-11T09:59:42.108 INFO:teuthology.orchestra.run.smithi040.stdout:Rotating logs: 2024-01-11T09:59:42.110 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-01-11T09:59:42.164 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-01-11T09:59:42.167 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-11T09:59:42.167 DEBUG:teuthology.orchestra.run.smithi040:> /usr/sbin/getenforce 2024-01-11T09:59:42.192 INFO:teuthology.orchestra.run.smithi040.stdout:Permissive 2024-01-11T09:59:42.192 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-01-11T09:59:42.227 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-01-11T09:59:42.228 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi040.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi134.front.sepia.ceph.com': 'permissive'} 2024-01-11T09:59:42.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T09:59:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:59:42.262 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T09:59:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T09:59:42.297 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-11T09:59:42.297 DEBUG:teuthology.orchestra.run.smithi040:> sudo /usr/sbin/setenforce permissive 2024-01-11T09:59:42.331 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-01-11T09:59:42.371 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-11T09:59:42.381 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-11T09:59:42.386 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-11T09:59:42.505 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-11T09:59:42.521 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-11T09:59:42.522 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi040.front.sepia.ceph.com,smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-11T10:04:06.401 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-01-11T10:04:06.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-01-11T10:04:06.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T10:04:06.499 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-01-11T10:04:06.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-01-11T10:04:06.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-11T10:04:06.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-11T10:04:06.668 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-01-11T10:04:06.743 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-01-11T10:04:06.744 INFO:teuthology.run_tasks:Running task clock... 2024-01-11T10:04:06.754 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-11T10:04:06.754 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T10:04:06.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T10:04:06.758 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-11T10:04:06.758 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T10:04:06.787 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-11T10:04:06.803 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-11T10:04:06.815 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-11T10:04:06.831 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-11T10:04:06.832 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: ntpd: command not found 2024-01-11T10:04:06.845 INFO:teuthology.orchestra.run.smithi040.stdout:506 Cannot talk to daemon 2024-01-11T10:04:06.861 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-01-11T10:04:06.862 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-11T10:04:06.875 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-01-11T10:04:06.878 INFO:teuthology.orchestra.run.smithi040.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-11T10:04:06.891 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-11T10:04:06.907 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-11T10:04:06.932 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-01-11T10:04:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T10:04:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-01-11T10:04:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.935 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.957 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-01-11T10:04:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T10:04:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-11T10:04:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T10:04:06.962 INFO:teuthology.run_tasks:Running task install... 2024-01-11T10:04:06.970 DEBUG:teuthology.task.install:project ceph 2024-01-11T10:04:06.970 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7'}} 2024-01-11T10:04:06.970 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7'} 2024-01-11T10:04:06.970 INFO:teuthology.task.install:Using flavor: crimson 2024-01-11T10:04:06.977 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-11T10:04:06.977 INFO:teuthology.task.install:extra packages: [] 2024-01-11T10:04:06.977 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7', 'tag': None, 'wait_for_package': False} 2024-01-11T10:04:06.978 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e3957a13c3b8938fe41053b13fb80161e1916b7 2024-01-11T10:04:06.979 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7', 'tag': None, 'wait_for_package': False} 2024-01-11T10:04:06.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e3957a13c3b8938fe41053b13fb80161e1916b7 2024-01-11T10:04:07.135 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/ 2024-01-11T10:04:07.135 INFO:teuthology.task.install.rpm:Package version is 19.0.0-635.g4e3957a1 2024-01-11T10:04:07.197 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/ 2024-01-11T10:04:07.197 INFO:teuthology.task.install.rpm:Package version is 19.0.0-635.g4e3957a1 2024-01-11T10:04:07.429 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-11T10:04:07.429 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:04:07.429 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-11T10:04:07.458 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-11T10:04:07.458 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3957a13c3b8938fe41053b13fb80161e1916b7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-11T10:04:07.496 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-multicore-msgr/4e3957a13c3b8938fe41053b13fb80161e1916b7/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-11T10:04:07.496 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:04:07.497 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-11T10:04:07.524 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-11T10:04:07.524 DEBUG:teuthology.orchestra.run.smithi040:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4e3957a13c3b8938fe41053b13fb80161e1916b7/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-11T10:04:07.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-11T10:04:07.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-11T10:04:07.606 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-11T10:04:07.671 DEBUG:teuthology.orchestra.run.smithi040:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-11T10:04:07.673 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-01-11T10:04:07.675 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-01-11T10:04:07.737 INFO:teuthology.orchestra.run.smithi040.stdout:check_obsoletes = 1 2024-01-11T10:04:07.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum clean all 2024-01-11T10:04:07.935 INFO:teuthology.orchestra.run.smithi134.stdout:73 files removed 2024-01-11T10:04:07.962 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-01-11T10:04:07.986 INFO:teuthology.orchestra.run.smithi040.stdout:73 files removed 2024-01-11T10:04:08.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-radosgw 2024-01-11T10:04:09.087 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 104 kB/s | 83 kB 00:00 2024-01-11T10:04:09.098 INFO:teuthology.orchestra.run.smithi040.stdout:ceph packages for x86_64 110 kB/s | 83 kB 00:00 2024-01-11T10:04:09.562 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2024-01-11T10:04:09.591 INFO:teuthology.orchestra.run.smithi040.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2024-01-11T10:04:09.938 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-01-11T10:04:09.975 INFO:teuthology.orchestra.run.smithi040.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-01-11T10:04:10.590 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 13 MB/s | 7.8 MB 00:00 2024-01-11T10:04:11.060 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - AppStream 7.4 MB/s | 7.8 MB 00:01 2024-01-11T10:04:12.190 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.5 MB 00:00 2024-01-11T10:04:12.727 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.5 MB 00:00 2024-01-11T10:04:13.751 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 7.9 MB 00:00 2024-01-11T10:04:13.889 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 7.9 MB 00:01 2024-01-11T10:04:15.740 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 24 MB/s | 18 MB 00:00 2024-01-11T10:04:16.705 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2024-01-11T10:04:20.208 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 5.9 MB 00:01 2024-01-11T10:04:21.795 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-01-11T10:04:21.809 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 15 kB 00:00 2024-01-11T10:04:24.787 INFO:teuthology.orchestra.run.smithi040.stdout:CentOS Stream 9 - Extras packages 8.1 kB/s | 15 kB 00:01 2024-01-11T10:04:26.183 INFO:teuthology.orchestra.run.smithi040.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-01-11T10:04:26.488 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 20 MB 00:04 2024-01-11T10:04:31.247 INFO:teuthology.orchestra.run.smithi040.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-01-11T10:04:31.367 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-01-11T10:04:33.819 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:04:33.819 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:04:33.820 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:04:33.821 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:04:33.822 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:04:33.823 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.824 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:04:33.825 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.826 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.827 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.828 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.838 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.839 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.840 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:04:33.841 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.842 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:04:33.843 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.843 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:04:33.843 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.843 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:04:33.843 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.843 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.844 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.845 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.846 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.847 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.848 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.849 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.849 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:04:33.849 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.849 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.849 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:04:33.849 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:04:34.321 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:04:34.329 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 8.4 M 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrading: 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: librados2 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 3.1 M 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 3.1 M 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 5.1 M 2024-01-11T10:04:34.333 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 18 M 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 25 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 751 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 497 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 4.5 M 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-01-11T10:04:34.334 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 45 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 130 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 161 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 320 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 298 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 99 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout:Install 27 Packages 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout:Upgrade 2 Packages 2024-01-11T10:04:34.335 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:04:34.336 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 53 M 2024-01-11T10:04:34.336 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 8.4 M 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 3.1 M 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 3.1 M 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-01-11T10:04:34.337 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 5.1 M 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 18 M 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 25 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 751 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 497 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 4.5 M 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-01-11T10:04:34.338 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 45 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 130 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 161 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 320 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 298 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 99 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout:Install 27 Packages 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-01-11T10:04:34.339 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:34.340 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 53 M 2024-01-11T10:04:34.340 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:04:35.302 INFO:teuthology.orchestra.run.smithi134.stdout:(1/29): ceph-base-19.0.0-635.g4e3957a1.el9.x86_ 8.0 MB/s | 5.1 MB 00:00 2024-01-11T10:04:35.352 INFO:teuthology.orchestra.run.smithi134.stdout:(2/29): ceph-selinux-19.0.0-635.g4e3957a1.el9.x 504 kB/s | 25 kB 00:00 2024-01-11T10:04:35.444 INFO:teuthology.orchestra.run.smithi134.stdout:(3/29): libcephfs2-19.0.0-635.g4e3957a1.el9.x86 8.0 MB/s | 751 kB 00:00 2024-01-11T10:04:35.528 INFO:teuthology.orchestra.run.smithi134.stdout:(4/29): libradosstriper1-19.0.0-635.g4e3957a1.e 5.8 MB/s | 497 kB 00:00 2024-01-11T10:04:36.103 INFO:teuthology.orchestra.run.smithi134.stdout:(5/29): librgw2-19.0.0-635.g4e3957a1.el9.x86_64 7.8 MB/s | 4.5 MB 00:00 2024-01-11T10:04:36.178 INFO:teuthology.orchestra.run.smithi134.stdout:(6/29): python3-ceph-argparse-19.0.0-635.g4e395 603 kB/s | 45 kB 00:00 2024-01-11T10:04:36.262 INFO:teuthology.orchestra.run.smithi134.stdout:(7/29): python3-ceph-common-19.0.0-635.g4e3957a 1.5 MB/s | 130 kB 00:00 2024-01-11T10:04:36.412 INFO:teuthology.orchestra.run.smithi134.stdout:(8/29): ceph-radosgw-19.0.0-635.g4e3957a1.el9.x 4.8 MB/s | 8.4 MB 00:01 2024-01-11T10:04:36.437 INFO:teuthology.orchestra.run.smithi134.stdout:(9/29): python3-cephfs-19.0.0-635.g4e3957a1.el9 921 kB/s | 161 kB 00:00 2024-01-11T10:04:36.487 INFO:teuthology.orchestra.run.smithi134.stdout:(10/29): python3-rados-19.0.0-635.g4e3957a1.el9 4.2 MB/s | 320 kB 00:00 2024-01-11T10:04:36.521 INFO:teuthology.orchestra.run.smithi134.stdout:(11/29): python3-rbd-19.0.0-635.g4e3957a1.el9.x 3.5 MB/s | 298 kB 00:00 2024-01-11T10:04:36.563 INFO:teuthology.orchestra.run.smithi134.stdout:(12/29): python3-rgw-19.0.0-635.g4e3957a1.el9.x 1.3 MB/s | 99 kB 00:00 2024-01-11T10:04:36.730 INFO:teuthology.orchestra.run.smithi134.stdout:(13/29): ceph-common-19.0.0-635.g4e3957a1.el9.x 8.5 MB/s | 18 MB 00:02 2024-01-11T10:04:37.080 INFO:teuthology.orchestra.run.smithi134.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 129 kB/s | 45 kB 00:00 2024-01-11T10:04:37.097 INFO:teuthology.orchestra.run.smithi134.stdout:(15/29): boost-program-options-1.75.0-8.el9.x86 185 kB/s | 107 kB 00:00 2024-01-11T10:04:37.106 INFO:teuthology.orchestra.run.smithi040.stdout:(1/29): ceph-base-19.0.0-635.g4e3957a1.el9.x86_ 2.0 MB/s | 5.1 MB 00:02 2024-01-11T10:04:37.147 INFO:teuthology.orchestra.run.smithi134.stdout:(16/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 274 kB/s | 160 kB 00:00 2024-01-11T10:04:37.190 INFO:teuthology.orchestra.run.smithi040.stdout:(2/29): ceph-selinux-19.0.0-635.g4e3957a1.el9.x 303 kB/s | 25 kB 00:00 2024-01-11T10:04:37.264 INFO:teuthology.orchestra.run.smithi134.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-01-11T10:04:37.373 INFO:teuthology.orchestra.run.smithi134.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 292 kB 00:00 2024-01-11T10:04:37.448 INFO:teuthology.orchestra.run.smithi040.stdout:(3/29): libcephfs2-19.0.0-635.g4e3957a1.el9.x86 2.8 MB/s | 751 kB 00:00 2024-01-11T10:04:37.465 INFO:teuthology.orchestra.run.smithi134.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-01-11T10:04:37.540 INFO:teuthology.orchestra.run.smithi040.stdout:(4/29): ceph-radosgw-19.0.0-635.g4e3957a1.el9.x 2.8 MB/s | 8.4 MB 00:03 2024-01-11T10:04:37.548 INFO:teuthology.orchestra.run.smithi134.stdout:(20/29): librdkafka-1.6.1-102.el9.x86_64.rpm 1.4 MB/s | 662 kB 00:00 2024-01-11T10:04:37.574 INFO:teuthology.orchestra.run.smithi134.stdout:(21/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-01-11T10:04:37.641 INFO:teuthology.orchestra.run.smithi134.stdout:(22/29): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-01-11T10:04:37.649 INFO:teuthology.orchestra.run.smithi040.stdout:(5/29): libradosstriper1-19.0.0-635.g4e3957a1.e 2.4 MB/s | 497 kB 00:00 2024-01-11T10:04:37.666 INFO:teuthology.orchestra.run.smithi134.stdout:(23/29): liboath-2.6.7-2.el9.x86_64.rpm 422 kB/s | 49 kB 00:00 2024-01-11T10:04:37.683 INFO:teuthology.orchestra.run.smithi134.stdout:(24/29): libunwind-1.6.2-1.el9.x86_64.rpm 624 kB/s | 67 kB 00:00 2024-01-11T10:04:37.725 INFO:teuthology.orchestra.run.smithi134.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-01-11T10:04:37.750 INFO:teuthology.orchestra.run.smithi134.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-01-11T10:04:37.757 INFO:teuthology.orchestra.run.smithi040.stdout:(6/29): python3-ceph-argparse-19.0.0-635.g4e395 417 kB/s | 45 kB 00:00 2024-01-11T10:04:37.792 INFO:teuthology.orchestra.run.smithi134.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-01-11T10:04:37.849 INFO:teuthology.orchestra.run.smithi040.stdout:(7/29): python3-ceph-common-19.0.0-635.g4e3957a 1.4 MB/s | 130 kB 00:00 2024-01-11T10:04:37.991 INFO:teuthology.orchestra.run.smithi040.stdout:(8/29): python3-cephfs-19.0.0-635.g4e3957a1.el9 1.1 MB/s | 161 kB 00:00 2024-01-11T10:04:38.559 INFO:teuthology.orchestra.run.smithi134.stdout:(28/29): librados2-19.0.0-635.g4e3957a1.el9.x86 3.8 MB/s | 3.1 MB 00:00 2024-01-11T10:04:38.583 INFO:teuthology.orchestra.run.smithi040.stdout:(9/29): librgw2-19.0.0-635.g4e3957a1.el9.x86_64 4.3 MB/s | 4.5 MB 00:01 2024-01-11T10:04:38.617 INFO:teuthology.orchestra.run.smithi134.stdout:(29/29): librbd1-19.0.0-635.g4e3957a1.el9.x86_6 3.6 MB/s | 3.1 MB 00:00 2024-01-11T10:04:38.621 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:04:38.621 INFO:teuthology.orchestra.run.smithi134.stdout:Total 12 MB/s | 53 MB 00:04 2024-01-11T10:04:38.694 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:04:38.717 INFO:teuthology.orchestra.run.smithi040.stdout:(10/29): python3-rbd-19.0.0-635.g4e3957a1.el9.x 2.2 MB/s | 298 kB 00:00 2024-01-11T10:04:38.732 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:04:38.732 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:04:38.792 INFO:teuthology.orchestra.run.smithi040.stdout:(11/29): python3-rgw-19.0.0-635.g4e3957a1.el9.x 1.3 MB/s | 99 kB 00:00 2024-01-11T10:04:38.909 INFO:teuthology.orchestra.run.smithi040.stdout:(12/29): python3-rados-19.0.0-635.g4e3957a1.el9 349 kB/s | 320 kB 00:00 2024-01-11T10:04:39.134 INFO:teuthology.orchestra.run.smithi040.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 312 kB/s | 107 kB 00:00 2024-01-11T10:04:39.160 INFO:teuthology.orchestra.run.smithi040.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 640 kB/s | 160 kB 00:00 2024-01-11T10:04:39.194 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:04:39.194 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:04:39.210 INFO:teuthology.orchestra.run.smithi040.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-01-11T10:04:39.294 INFO:teuthology.orchestra.run.smithi040.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-01-11T10:04:39.335 INFO:teuthology.orchestra.run.smithi040.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-01-11T10:04:39.353 INFO:teuthology.orchestra.run.smithi040.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 720 kB/s | 42 kB 00:00 2024-01-11T10:04:39.545 INFO:teuthology.orchestra.run.smithi040.stdout:(19/29): ceph-common-19.0.0-635.g4e3957a1.el9.x 3.5 MB/s | 18 MB 00:05 2024-01-11T10:04:39.670 INFO:teuthology.orchestra.run.smithi040.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-01-11T10:04:39.778 INFO:teuthology.orchestra.run.smithi040.stdout:(21/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2024-01-11T10:04:39.795 INFO:teuthology.orchestra.run.smithi040.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 387 kB/s | 49 kB 00:00 2024-01-11T10:04:39.837 INFO:teuthology.orchestra.run.smithi040.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-01-11T10:04:39.912 INFO:teuthology.orchestra.run.smithi040.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-01-11T10:04:40.021 INFO:teuthology.orchestra.run.smithi040.stdout:(25/29): libarrow-9.0.0-10.el9.x86_64.rpm 6.6 MB/s | 4.4 MB 00:00 2024-01-11T10:04:40.046 INFO:teuthology.orchestra.run.smithi040.stdout:(26/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-01-11T10:04:40.122 INFO:teuthology.orchestra.run.smithi040.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-01-11T10:04:40.532 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:04:40.638 INFO:teuthology.orchestra.run.smithi040.stdout:(28/29): librbd1-19.0.0-635.g4e3957a1.el9.x86_6 5.2 MB/s | 3.1 MB 00:00 2024-01-11T10:04:40.780 INFO:teuthology.orchestra.run.smithi040.stdout:(29/29): librados2-19.0.0-635.g4e3957a1.el9.x86 4.1 MB/s | 3.1 MB 00:00 2024-01-11T10:04:40.782 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:04:40.783 INFO:teuthology.orchestra.run.smithi040.stdout:Total 8.2 MB/s | 53 MB 00:06 2024-01-11T10:04:40.857 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:04:40.885 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-11T10:04:40.895 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:04:40.895 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:04:40.977 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 2/31 2024-01-11T10:04:41.070 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 2/31 2024-01-11T10:04:41.159 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 3/31 2024-01-11T10:04:41.187 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 4/31 2024-01-11T10:04:41.255 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 4/31 2024-01-11T10:04:41.283 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-11T10:04:41.308 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-11T10:04:41.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-11T10:04:41.368 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:04:41.368 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:04:41.381 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-635.g4e3957a1.el9.x 8/31 2024-01-11T10:04:41.452 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 9/31 2024-01-11T10:04:41.474 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 10/31 2024-01-11T10:04:41.562 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 10/31 2024-01-11T10:04:41.598 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-11T10:04:41.705 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-11T10:04:41.741 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-11T10:04:41.781 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-11T10:04:41.813 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-11T10:04:41.850 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-11T10:04:41.884 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-11T10:04:42.239 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-11T10:04:42.271 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 19/31 2024-01-11T10:04:42.348 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 19/31 2024-01-11T10:04:42.382 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 20/31 2024-01-11T10:04:42.439 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-11T10:04:42.858 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:04:42.868 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-11T10:04:42.892 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 23/31 2024-01-11T10:04:42.969 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 23/31 2024-01-11T10:04:43.041 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 24/31 2024-01-11T10:04:43.070 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-635.g4e3957a1.el9.x86 25/31 2024-01-11T10:04:43.307 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-11T10:04:43.341 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 2/31 2024-01-11T10:04:43.417 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 2/31 2024-01-11T10:04:43.517 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 3/31 2024-01-11T10:04:43.551 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 4/31 2024-01-11T10:04:43.634 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 4/31 2024-01-11T10:04:43.663 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-11T10:04:43.689 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-11T10:04:43.718 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-11T10:04:43.821 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-argparse-2:19.0.0-635.g4e3957a1.el9.x 8/31 2024-01-11T10:04:43.926 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 9/31 2024-01-11T10:04:43.954 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 10/31 2024-01-11T10:04:44.043 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 10/31 2024-01-11T10:04:44.079 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-11T10:04:44.188 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-11T10:04:44.221 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-11T10:04:44.262 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-11T10:04:44.302 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-11T10:04:44.331 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-11T10:04:44.364 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-11T10:04:44.709 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-11T10:04:44.744 INFO:teuthology.orchestra.run.smithi040.stdout: Upgrading : librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 19/31 2024-01-11T10:04:44.827 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 19/31 2024-01-11T10:04:44.857 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 20/31 2024-01-11T10:04:44.893 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-11T10:04:45.096 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 26/31 2024-01-11T10:04:45.126 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 26/31 2024-01-11T10:04:45.347 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-11T10:04:45.371 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 23/31 2024-01-11T10:04:45.448 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 23/31 2024-01-11T10:04:45.521 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 24/31 2024-01-11T10:04:45.548 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ceph-common-2:19.0.0-635.g4e3957a1.el9.x86 25/31 2024-01-11T10:04:45.678 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 26/31 2024-01-11T10:04:45.720 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 27/31 2024-01-11T10:04:45.809 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 27/31 2024-01-11T10:04:45.809 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-11T10:04:45.809 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-11T10:04:45.809 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:45.830 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 28/31 2024-01-11T10:04:47.674 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 26/31 2024-01-11T10:04:47.704 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 26/31 2024-01-11T10:04:48.287 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 26/31 2024-01-11T10:04:48.332 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 27/31 2024-01-11T10:04:48.422 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 27/31 2024-01-11T10:04:48.423 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-11T10:04:48.423 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-11T10:04:48.423 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:04:48.458 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 28/31 2024-01-11T10:04:55.353 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 28/31 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-01-11T10:04:55.354 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:56.190 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 29/31 2024-01-11T10:04:56.214 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 29/31 2024-01-11T10:04:56.215 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:04:56.215 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-11T10:04:56.215 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-11T10:04:56.215 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-11T10:04:56.215 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:56.216 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-11T10:04:56.284 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-11T10:04:56.284 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 1/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 2/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 3/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 4/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 5/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 6/31 2024-01-11T10:04:57.439 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 7/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-635.g4e3957a1.el9.x 8/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-635.g4e3957a1.el9.x86 9/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 10/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 11/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 12/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 13/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-11T10:04:57.440 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 28/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-11T10:04:57.441 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 30/31 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.517 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-11T10:04:58.518 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:58.519 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:04:58.559 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 28/31 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /sys 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /proc 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /mnt 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /var/tmp 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /home 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /root 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout:skipping the directory /tmp 2024-01-11T10:04:58.560 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:04:58.870 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-01-11T10:04:59.435 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 29/31 2024-01-11T10:04:59.459 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 29/31 2024-01-11T10:04:59.460 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:04:59.460 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-11T10:04:59.460 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-11T10:04:59.460 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-11T10:04:59.460 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:04:59.463 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-11T10:04:59.464 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:28 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:04:59.547 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-11T10:04:59.547 INFO:teuthology.orchestra.run.smithi040.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:04:59.754 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:04:59.755 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:04:59.756 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.757 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:04:59.758 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:04:59.759 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.760 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.761 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.762 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.763 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:04:59.763 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.763 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.763 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.763 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:04:59.829 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:04:59.830 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:04:59.830 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:04:59.830 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:04:59.830 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 119 M 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: libasan x86_64 11.4.1-3.el9 appstream 418 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 158 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:04:59.831 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:04:59.832 INFO:teuthology.orchestra.run.smithi134.stdout:Install 10 Packages 2024-01-11T10:04:59.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:04:59.832 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 122 M 2024-01-11T10:04:59.832 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.1 G 2024-01-11T10:04:59.832 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:05:00.668 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-11T10:05:00.668 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 1/31 2024-01-11T10:05:00.668 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 2/31 2024-01-11T10:05:00.668 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 3/31 2024-01-11T10:05:00.668 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 4/31 2024-01-11T10:05:00.668 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 5/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 6/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 7/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-argparse-2:19.0.0-635.g4e3957a1.el9.x 8/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ceph-common-2:19.0.0-635.g4e3957a1.el9.x86 9/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 10/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 11/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 12/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 13/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-11T10:05:00.669 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 28/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-11T10:05:00.670 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 30/31 2024-01-11T10:05:00.787 INFO:teuthology.orchestra.run.smithi134.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 285 kB/s | 187 kB 00:00 2024-01-11T10:05:00.821 INFO:teuthology.orchestra.run.smithi134.stdout:(2/10): libcephsqlite-19.0.0-635.g4e3957a1.el9. 229 kB/s | 158 kB 00:00 2024-01-11T10:05:01.013 INFO:teuthology.orchestra.run.smithi134.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 967 kB/s | 218 kB 00:00 2024-01-11T10:05:01.439 INFO:teuthology.orchestra.run.smithi134.stdout:(4/10): libasan-11.4.1-3.el9.x86_64.rpm 678 kB/s | 418 kB 00:00 2024-01-11T10:05:01.539 INFO:teuthology.orchestra.run.smithi134.stdout:(5/10): libubsan-11.4.1-3.el9.x86_64.rpm 356 kB/s | 187 kB 00:00 2024-01-11T10:05:01.581 INFO:teuthology.orchestra.run.smithi134.stdout:(6/10): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 305 kB 00:00 2024-01-11T10:05:01.656 INFO:teuthology.orchestra.run.smithi134.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 544 kB/s | 64 kB 00:00 2024-01-11T10:05:01.744 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-11T10:05:01.744 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout:Upgraded: 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-radosgw-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-selinux-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-11T10:05:01.745 INFO:teuthology.orchestra.run.smithi040.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: librgw2-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-argparse-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.746 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: re2-1:20211101-3.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:01.747 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:05:02.032 INFO:teuthology.orchestra.run.smithi134.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 348 kB/s | 122 kB 00:00 2024-01-11T10:05:02.057 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-test 2024-01-11T10:05:02.157 INFO:teuthology.orchestra.run.smithi134.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.2 MB/s | 1.3 MB 00:00 2024-01-11T10:05:02.654 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:00:31 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:05:02.958 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:05:02.958 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:05:02.958 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:05:02.958 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.958 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:05:02.959 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.960 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.961 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:05:02.962 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.963 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.964 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.965 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.966 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.967 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.967 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:05:02.967 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.967 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.967 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:05:02.967 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:05:03.029 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:05:03.029 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 119 M 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: libasan x86_64 11.4.1-3.el9 appstream 418 k 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 158 k 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma x86_64 6.9.6-1.el9.5 baseos 218 k 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-01-11T10:05:03.030 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout:Install 10 Packages 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:03.031 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 122 M 2024-01-11T10:05:03.032 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.1 G 2024-01-11T10:05:03.032 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:05:03.850 INFO:teuthology.orchestra.run.smithi040.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 607 kB/s | 187 kB 00:00 2024-01-11T10:05:03.933 INFO:teuthology.orchestra.run.smithi040.stdout:(2/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.6 MB/s | 218 kB 00:00 2024-01-11T10:05:04.034 INFO:teuthology.orchestra.run.smithi040.stdout:(3/10): libasan-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 418 kB 00:00 2024-01-11T10:05:04.059 INFO:teuthology.orchestra.run.smithi040.stdout:(4/10): libcephsqlite-19.0.0-635.g4e3957a1.el9. 306 kB/s | 158 kB 00:00 2024-01-11T10:05:04.093 INFO:teuthology.orchestra.run.smithi040.stdout:(5/10): libubsan-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-01-11T10:05:04.152 INFO:teuthology.orchestra.run.smithi040.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-01-11T10:05:04.285 INFO:teuthology.orchestra.run.smithi040.stdout:(7/10): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 305 kB 00:00 2024-01-11T10:05:04.711 INFO:teuthology.orchestra.run.smithi040.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 287 kB/s | 122 kB 00:00 2024-01-11T10:05:05.519 INFO:teuthology.orchestra.run.smithi040.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 941 kB/s | 1.3 MB 00:01 2024-01-11T10:05:19.723 INFO:teuthology.orchestra.run.smithi134.stdout:(10/10): ceph-test-19.0.0-635.g4e3957a1.el9.x86 6.1 MB/s | 119 MB 00:19 2024-01-11T10:05:19.725 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:05:19.725 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.1 MB/s | 122 MB 00:19 2024-01-11T10:05:19.797 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:05:19.818 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:05:19.818 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:05:20.118 INFO:teuthology.orchestra.run.smithi040.stdout:(10/10): ceph-test-19.0.0-635.g4e3957a1.el9.x86 7.2 MB/s | 119 MB 00:16 2024-01-11T10:05:20.119 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:05:20.119 INFO:teuthology.orchestra.run.smithi040.stdout:Total 7.1 MB/s | 122 MB 00:17 2024-01-11T10:05:20.195 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:05:20.217 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:05:20.217 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:05:20.653 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:05:20.653 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:05:21.082 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:05:21.082 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:05:21.778 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:05:21.869 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-11T10:05:21.903 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-11T10:05:21.948 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2024-01-11T10:05:21.981 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2024-01-11T10:05:22.014 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/10 2024-01-11T10:05:22.049 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libasan-11.4.1-3.el9.x86_64 6/10 2024-01-11T10:05:22.081 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 7/10 2024-01-11T10:05:22.123 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 8/10 2024-01-11T10:05:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 9/10 2024-01-11T10:05:22.247 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:05:22.321 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-11T10:05:22.367 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-11T10:05:22.412 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2024-01-11T10:05:22.443 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2024-01-11T10:05:22.478 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/10 2024-01-11T10:05:22.521 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libasan-11.4.1-3.el9.x86_64 6/10 2024-01-11T10:05:22.561 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 7/10 2024-01-11T10:05:22.612 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : jq-1.6-16.el9.x86_64 8/10 2024-01-11T10:05:22.639 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 9/10 2024-01-11T10:05:38.640 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 9/10 2024-01-11T10:05:38.715 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 10/10 2024-01-11T10:05:39.997 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 9/10 2024-01-11T10:05:40.074 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 10/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 10/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 1/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 2/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 5/10 2024-01-11T10:05:46.845 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 6/10 2024-01-11T10:05:46.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2024-01-11T10:05:46.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2024-01-11T10:05:46.846 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-11T10:05:47.315 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:05:47.316 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:05:47.619 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-01-11T10:05:48.238 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:17 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 10/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 1/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 2/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 5/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 6/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2024-01-11T10:05:48.356 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-11T10:05:48.526 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:05:48.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:05:48.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:05:48.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:05:48.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.533 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.534 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.535 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.536 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.536 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:05:48.536 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.536 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.536 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.536 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:05:48.599 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 6.6 k 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 2.0 M 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 1.5 M 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 244 k 2024-01-11T10:05:48.602 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 4.8 M 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 117 M 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-11T10:05:48.603 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-7.el9 baseos 124 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-01-11T10:05:48.604 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-01-11T10:05:48.605 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:05:48.606 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 130 M 2024-01-11T10:05:48.606 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.4 G 2024-01-11T10:05:48.606 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:05:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-11T10:05:48.824 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:48.824 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-test-2:19.0.0-635.g4e3957a1.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: libcephsqlite-2:19.0.0-635.g4e3957a1.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:48.825 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:05:49.135 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph 2024-01-11T10:05:49.242 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-635.g4e3957a1.el9.x86_64.rp 41 kB/s | 6.6 kB 00:00 2024-01-11T10:05:49.426 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mgr-19.0.0-635.g4e3957a1.el9.x86_6 4.3 MB/s | 1.5 MB 00:00 2024-01-11T10:05:49.735 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mon-19.0.0-635.g4e3957a1.el9.x86_6 9.9 MB/s | 4.8 MB 00:00 2024-01-11T10:05:49.736 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:01:18 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:05:49.801 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-635.g4e395 3.6 MB/s | 244 kB 00:00 2024-01-11T10:05:50.044 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:05:50.045 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:05:50.045 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:05:50.045 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.045 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:05:50.045 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.046 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.047 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:05:50.048 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.049 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:05:50.050 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.051 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.052 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.053 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.054 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:05:50.126 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: ceph x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 6.6 k 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 2.0 M 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 1.5 M 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 244 k 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 4.8 M 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 117 M 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-01-11T10:05:50.129 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-01-11T10:05:50.130 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests noarch 2.25.1-7.el9 baseos 124 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-11T10:05:50.131 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:50.132 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:05:50.132 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:05:50.132 INFO:teuthology.orchestra.run.smithi040.stdout:Install 36 Packages 2024-01-11T10:05:50.132 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:05:50.132 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 130 M 2024-01-11T10:05:50.132 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.4 G 2024-01-11T10:05:50.133 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:05:50.427 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 405 kB/s | 253 kB 00:00 2024-01-11T10:05:50.852 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 2.9 MB/s | 1.2 MB 00:00 2024-01-11T10:05:50.961 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 986 kB/s | 106 kB 00:00 2024-01-11T10:05:51.086 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-01-11T10:05:51.203 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): ceph-mds-19.0.0-635.g4e3957a1.el9.x86_6 981 kB/s | 2.0 MB 00:02 2024-01-11T10:05:51.221 INFO:teuthology.orchestra.run.smithi040.stdout:(1/36): ceph-19.0.0-635.g4e3957a1.el9.x86_64.rp 17 kB/s | 6.6 kB 00:00 2024-01-11T10:05:51.228 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-requests-2.25.1-7.el9.noarch.r 876 kB/s | 124 kB 00:00 2024-01-11T10:05:51.495 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 706 kB/s | 188 kB 00:00 2024-01-11T10:05:51.554 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-01-11T10:05:51.604 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 692 kB/s | 35 kB 00:00 2024-01-11T10:05:51.671 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-pytz-2021.1-5.el9.noarch.rpm 767 kB/s | 51 kB 00:00 2024-01-11T10:05:51.738 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-toml-0.10.2-6.el9.noarch.rpm 624 kB/s | 42 kB 00:00 2024-01-11T10:05:51.822 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-01-11T10:05:51.922 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): python3-urllib3-1.26.5-5.el9.noarch.rp 299 kB/s | 215 kB 00:00 2024-01-11T10:05:51.947 INFO:teuthology.orchestra.run.smithi040.stdout:(2/36): ceph-mgr-19.0.0-635.g4e3957a1.el9.x86_6 1.3 MB/s | 1.5 MB 00:01 2024-01-11T10:05:51.980 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 953 kB/s | 151 kB 00:00 2024-01-11T10:05:52.023 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-01-11T10:05:52.048 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-01-11T10:05:52.090 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 257 kB/s | 11 kB 00:00 2024-01-11T10:05:52.132 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-01-11T10:05:52.166 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-01-11T10:05:52.191 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 394 kB/s | 23 kB 00:00 2024-01-11T10:05:52.224 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-01-11T10:05:52.251 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-01-11T10:05:52.283 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 788 kB/s | 46 kB 00:00 2024-01-11T10:05:52.309 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-01-11T10:05:52.367 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-01-11T10:05:52.388 INFO:teuthology.orchestra.run.smithi040.stdout:(3/36): ceph-mds-19.0.0-635.g4e3957a1.el9.x86_6 1.3 MB/s | 2.0 MB 00:01 2024-01-11T10:05:52.401 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-01-11T10:05:52.426 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-01-11T10:05:52.452 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-01-11T10:05:52.493 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-01-11T10:05:52.530 INFO:teuthology.orchestra.run.smithi040.stdout:(4/36): ceph-mgr-modules-core-19.0.0-635.g4e395 1.7 MB/s | 244 kB 00:00 2024-01-11T10:05:52.535 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-01-11T10:05:52.561 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-01-11T10:05:52.731 INFO:teuthology.orchestra.run.smithi040.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-01-11T10:05:52.931 INFO:teuthology.orchestra.run.smithi040.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 6.1 MB/s | 1.2 MB 00:00 2024-01-11T10:05:52.981 INFO:teuthology.orchestra.run.smithi040.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-01-11T10:05:53.040 INFO:teuthology.orchestra.run.smithi040.stdout:(8/36): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-01-11T10:05:53.107 INFO:teuthology.orchestra.run.smithi040.stdout:(9/36): python3-requests-2.25.1-7.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-01-11T10:05:53.165 INFO:teuthology.orchestra.run.smithi040.stdout:(10/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-01-11T10:05:53.682 INFO:teuthology.orchestra.run.smithi040.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 365 kB/s | 188 kB 00:00 2024-01-11T10:05:54.124 INFO:teuthology.orchestra.run.smithi040.stdout:(12/36): python3-mako-1.1.4-6.el9.noarch.rpm 390 kB/s | 172 kB 00:00 2024-01-11T10:05:54.391 INFO:teuthology.orchestra.run.smithi040.stdout:(13/36): python3-markupsafe-1.1.1-12.el9.x86_64 130 kB/s | 35 kB 00:00 2024-01-11T10:05:54.641 INFO:teuthology.orchestra.run.smithi040.stdout:(14/36): ceph-mon-19.0.0-635.g4e3957a1.el9.x86_ 1.4 MB/s | 4.8 MB 00:03 2024-01-11T10:05:54.733 INFO:teuthology.orchestra.run.smithi040.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 149 kB/s | 51 kB 00:00 2024-01-11T10:05:54.992 INFO:teuthology.orchestra.run.smithi040.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-01-11T10:05:55.009 INFO:teuthology.orchestra.run.smithi040.stdout:(17/36): python3-toml-0.10.2-6.el9.noarch.rpm 113 kB/s | 42 kB 00:00 2024-01-11T10:05:55.084 INFO:teuthology.orchestra.run.smithi040.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-01-11T10:05:55.134 INFO:teuthology.orchestra.run.smithi040.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-01-11T10:05:55.160 INFO:teuthology.orchestra.run.smithi040.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-01-11T10:05:55.193 INFO:teuthology.orchestra.run.smithi040.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 323 kB/s | 11 kB 00:00 2024-01-11T10:05:55.219 INFO:teuthology.orchestra.run.smithi040.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-01-11T10:05:55.236 INFO:teuthology.orchestra.run.smithi040.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-01-11T10:05:55.252 INFO:teuthology.orchestra.run.smithi040.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 693 kB/s | 23 kB 00:00 2024-01-11T10:05:55.269 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): ceph-osd-19.0.0-635.g4e3957a1.el9.x86_ 20 MB/s | 117 MB 00:05 2024-01-11T10:05:55.270 INFO:teuthology.orchestra.run.smithi040.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 579 kB/s | 19 kB 00:00 2024-01-11T10:05:55.271 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:05:55.271 INFO:teuthology.orchestra.run.smithi134.stdout:Total 19 MB/s | 130 MB 00:06 2024-01-11T10:05:55.295 INFO:teuthology.orchestra.run.smithi040.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 467 kB/s | 20 kB 00:00 2024-01-11T10:05:55.312 INFO:teuthology.orchestra.run.smithi040.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-01-11T10:05:55.337 INFO:teuthology.orchestra.run.smithi040.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-01-11T10:05:55.371 INFO:teuthology.orchestra.run.smithi040.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-01-11T10:05:55.388 INFO:teuthology.orchestra.run.smithi040.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-01-11T10:05:55.410 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:05:55.413 INFO:teuthology.orchestra.run.smithi040.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-01-11T10:05:55.430 INFO:teuthology.orchestra.run.smithi040.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 859 kB/s | 36 kB 00:00 2024-01-11T10:05:55.448 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:05:55.449 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:05:55.464 INFO:teuthology.orchestra.run.smithi040.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-01-11T10:05:55.489 INFO:teuthology.orchestra.run.smithi040.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 7.1 MB/s | 427 kB 00:00 2024-01-11T10:05:55.514 INFO:teuthology.orchestra.run.smithi040.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-01-11T10:05:56.458 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:05:56.459 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:05:58.371 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:05:58.432 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-11T10:05:58.466 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-11T10:05:58.499 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-11T10:05:58.571 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-11T10:05:58.623 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-11T10:05:58.658 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-11T10:05:58.746 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-11T10:05:58.828 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-11T10:05:58.867 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-11T10:05:58.907 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-11T10:05:58.933 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-11T10:05:58.961 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-01-11T10:05:58.993 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-01-11T10:05:59.034 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-01-11T10:05:59.068 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-01-11T10:05:59.103 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-01-11T10:05:59.141 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-01-11T10:05:59.213 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-01-11T10:05:59.349 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-01-11T10:05:59.424 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-01-11T10:05:59.477 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-01-11T10:05:59.516 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/36 2024-01-11T10:05:59.565 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-01-11T10:05:59.620 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-01-11T10:05:59.812 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-01-11T10:05:59.870 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-01-11T10:05:59.940 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-01-11T10:06:00.028 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-01-11T10:06:00.061 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-01-11T10:06:00.181 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-01-11T10:06:00.348 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-635.g4e3957a1.el9.n 31/36 2024-01-11T10:06:00.376 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 32/36 2024-01-11T10:06:00.401 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 32/36 2024-01-11T10:06:00.401 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:00.401 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-11T10:06:00.401 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-11T10:06:00.402 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-11T10:06:00.402 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:00.906 INFO:teuthology.orchestra.run.smithi040.stdout:(36/36): ceph-osd-19.0.0-635.g4e3957a1.el9.x86_ 13 MB/s | 117 MB 00:08 2024-01-11T10:06:00.908 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:00.908 INFO:teuthology.orchestra.run.smithi040.stdout:Total 12 MB/s | 130 MB 00:10 2024-01-11T10:06:01.046 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:06:01.087 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:06:01.087 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:06:02.145 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:06:02.146 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:06:04.104 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:06:04.173 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-11T10:06:04.206 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-11T10:06:04.234 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-11T10:06:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-11T10:06:04.358 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-11T10:06:04.391 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-11T10:06:04.488 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-11T10:06:04.565 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-11T10:06:04.611 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-11T10:06:04.646 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-11T10:06:04.675 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-11T10:06:04.710 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-01-11T10:06:04.751 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-01-11T10:06:04.793 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-01-11T10:06:04.835 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-01-11T10:06:04.890 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-01-11T10:06:04.928 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-01-11T10:06:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-01-11T10:06:05.160 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-01-11T10:06:05.236 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-01-11T10:06:05.287 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-01-11T10:06:05.326 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/36 2024-01-11T10:06:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-01-11T10:06:05.457 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-01-11T10:06:05.661 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-01-11T10:06:05.705 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-01-11T10:06:05.787 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-01-11T10:06:05.877 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-01-11T10:06:05.930 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-01-11T10:06:06.049 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-01-11T10:06:06.219 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-635.g4e3957a1.el9.n 31/36 2024-01-11T10:06:06.247 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 32/36 2024-01-11T10:06:06.272 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 32/36 2024-01-11T10:06:06.272 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:06.272 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-11T10:06:06.272 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-11T10:06:06.272 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-11T10:06:06.272 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:18.758 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 33/36 2024-01-11T10:06:18.784 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 33/36 2024-01-11T10:06:18.785 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:18.785 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-11T10:06:18.785 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-11T10:06:18.785 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-11T10:06:18.785 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:19.242 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 34/36 2024-01-11T10:06:19.265 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 34/36 2024-01-11T10:06:19.265 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:19.265 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-11T10:06:19.266 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-11T10:06:19.266 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-11T10:06:19.266 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:19.468 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 35/36 2024-01-11T10:06:19.491 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 35/36 2024-01-11T10:06:19.491 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:19.491 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-11T10:06:19.491 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-11T10:06:19.491 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-11T10:06:19.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:19.565 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 36/36 2024-01-11T10:06:25.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 33/36 2024-01-11T10:06:25.700 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 33/36 2024-01-11T10:06:25.700 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:25.700 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-11T10:06:25.700 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-11T10:06:25.700 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-11T10:06:25.700 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:26.182 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 34/36 2024-01-11T10:06:26.204 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 34/36 2024-01-11T10:06:26.205 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:26.205 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-11T10:06:26.205 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-11T10:06:26.205 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-11T10:06:26.205 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:26.415 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 35/36 2024-01-11T10:06:26.438 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 35/36 2024-01-11T10:06:26.439 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:26.439 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-11T10:06:26.439 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-11T10:06:26.439 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-11T10:06:26.439 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:26.508 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 36/36 2024-01-11T10:06:29.183 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 36/36 2024-01-11T10:06:29.184 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 1/36 2024-01-11T10:06:29.184 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 2/36 2024-01-11T10:06:29.184 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 3/36 2024-01-11T10:06:29.184 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 4/36 2024-01-11T10:06:29.184 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 5/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-635.g4e3957a1.el9.n 6/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 11/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-01-11T10:06:29.185 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-11T10:06:29.186 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-11T10:06:29.187 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-11T10:06:30.222 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-11T10:06:30.222 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:30.222 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-11T10:06:30.223 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-11T10:06:30.224 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:30.225 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:06:30.479 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-01-11T10:06:31.074 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:00 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.360 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:06:31.361 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:06:31.362 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.363 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:06:31.364 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.365 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.366 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:31.367 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.368 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:31.370 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:06:31.425 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:06:31.426 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:06:31.426 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:06:31.511 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-01-11T10:06:32.107 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:01 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:32.393 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:06:32.393 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.394 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:06:32.395 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:06:32.396 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:06:32.397 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.398 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.399 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.400 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.401 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.402 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.403 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:32.459 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 742 k 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 742 k 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 748 k 2024-01-11T10:06:32.460 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:06:32.982 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-635.g4e3957a1.el9.noarch.rpm 1.4 MB/s | 742 kB 00:00 2024-01-11T10:06:32.982 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:32.982 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.4 MB/s | 742 kB 00:00 2024-01-11T10:06:32.983 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:06:32.988 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:06:32.988 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:06:32.993 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:06:32.994 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:06:33.066 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:06:33.481 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:33.570 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:33.933 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:34.288 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:34.288 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:34.288 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:06:34.288 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:34.288 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:34.289 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:06:34.525 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-01-11T10:06:35.134 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:04 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:35.424 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:06:35.424 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:06:35.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:06:35.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:06:35.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:35.432 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.433 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:35.490 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 147 k 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:35.491 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:06:35.492 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:35.492 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 147 k 2024-01-11T10:06:35.492 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 481 k 2024-01-11T10:06:35.492 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:06:35.719 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-635.g4e3957a 647 kB/s | 147 kB 00:00 2024-01-11T10:06:35.720 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:35.720 INFO:teuthology.orchestra.run.smithi134.stdout:Total 642 kB/s | 147 kB 00:00 2024-01-11T10:06:35.720 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:06:35.727 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:06:35.728 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:06:35.780 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:06:35.780 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:06:35.922 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:06:35.987 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.e 1/1 2024-01-11T10:06:36.011 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.e 1/1 2024-01-11T10:06:36.011 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:36.011 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-11T10:06:36.011 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 36/36 2024-01-11T10:06:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 1/36 2024-01-11T10:06:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 2/36 2024-01-11T10:06:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 3/36 2024-01-11T10:06:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 4/36 2024-01-11T10:06:36.296 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 5/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-635.g4e3957a1.el9.n 6/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 11/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-01-11T10:06:36.297 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-11T10:06:36.298 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-11T10:06:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-11T10:06:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-11T10:06:36.299 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-11T10:06:36.981 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.e 1/1 2024-01-11T10:06:36.981 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:36.981 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:06:36.981 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:36.981 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:36.981 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:06:37.196 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mds-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:37.400 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mon-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-osd-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: lua-5.4.4-4.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-11T10:06:37.401 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ply-3.11-14.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-11T10:06:37.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-11T10:06:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-11T10:06:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-11T10:06:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-11T10:06:37.403 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:37.403 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:06:37.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-base 2024-01-11T10:06:37.803 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:06 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:38.090 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.091 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:06:38.092 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.093 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.094 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.095 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.096 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.097 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.098 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.099 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.100 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:06:38.157 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:06:38.158 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:06:38.158 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:06:38.244 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-01-11T10:06:38.269 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:07 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:38.571 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:06:38.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:38.581 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-base-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:06:38.640 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:06:38.641 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:06:38.641 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:06:38.715 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install cephadm 2024-01-11T10:06:38.839 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:07 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:06:39.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:06:39.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:06:39.129 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.130 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.132 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.133 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.134 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.135 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.136 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:39.136 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.136 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.136 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.136 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.190 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:06:39.190 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:39.190 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 1.7 M 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 28 k 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 15 k 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout:Install 5 Packages 2024-01-11T10:06:39.191 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:39.192 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-01-11T10:06:39.192 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 8.4 M 2024-01-11T10:06:39.192 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:06:39.324 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:08 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:39.439 INFO:teuthology.orchestra.run.smithi134.stdout:(1/5): ceph-grafana-dashboards-19.0.0-635.g4e39 144 kB/s | 28 kB 00:00 2024-01-11T10:06:39.507 INFO:teuthology.orchestra.run.smithi134.stdout:(2/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 461 kB/s | 31 kB 00:00 2024-01-11T10:06:39.532 INFO:teuthology.orchestra.run.smithi134.stdout:(3/5): ceph-prometheus-alerts-19.0.0-635.g4e395 53 kB/s | 15 kB 00:00 2024-01-11T10:06:39.598 INFO:teuthology.orchestra.run.smithi134.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.625 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.626 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.627 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.628 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.629 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:06:39.630 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.631 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.632 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.633 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.634 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:39.635 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.635 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.635 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.635 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:39.693 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:06:39.693 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:39.693 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:06:39.693 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 742 k 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 742 k 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 748 k 2024-01-11T10:06:39.694 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:06:40.665 INFO:teuthology.orchestra.run.smithi134.stdout:(5/5): ceph-mgr-dashboard-19.0.0-635.g4e3957a1. 1.2 MB/s | 1.7 MB 00:01 2024-01-11T10:06:40.668 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:40.668 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-01-11T10:06:40.668 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:06:40.674 INFO:teuthology.orchestra.run.smithi040.stdout:cephadm-19.0.0-635.g4e3957a1.el9.noarch.rpm 757 kB/s | 742 kB 00:00 2024-01-11T10:06:40.675 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:40.675 INFO:teuthology.orchestra.run.smithi040.stdout:Total 756 kB/s | 742 kB 00:00 2024-01-11T10:06:40.675 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:06:40.676 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:06:40.676 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:06:40.681 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:06:40.681 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:06:40.687 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:06:40.688 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:06:40.746 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:06:40.746 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:06:40.762 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:06:40.999 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:06:41.096 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-11T10:06:41.136 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-11T10:06:41.149 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:41.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-635.g4e3957a1.el9.no 3/5 2024-01-11T10:06:41.242 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:41.401 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-635.g4e3957a1.el9.n 4/5 2024-01-11T10:06:41.435 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 5/5 2024-01-11T10:06:41.646 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:41.964 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:06:41.964 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:41.964 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:06:41.964 INFO:teuthology.orchestra.run.smithi040.stdout: cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:41.964 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:41.964 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:06:41.968 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 5/5 2024-01-11T10:06:41.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-635.g4e3957a1.el9.n 1/5 2024-01-11T10:06:41.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 2/5 2024-01-11T10:06:41.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-635.g4e3957a1.el9.no 3/5 2024-01-11T10:06:41.969 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-11T10:06:42.177 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-immutable-object-cache 2024-01-11T10:06:42.287 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-11T10:06:42.287 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:42.287 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:42.288 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:06:42.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-11T10:06:42.787 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:11 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:43.089 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:06:43.090 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:06:43.091 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:06:43.092 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:06:43.093 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:06:43.094 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.095 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.096 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.097 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.098 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.127 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:12 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 147 k 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:43.159 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:06:43.160 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:43.160 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:06:43.160 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:43.160 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 147 k 2024-01-11T10:06:43.160 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 481 k 2024-01-11T10:06:43.160 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.422 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.423 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.424 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:06:43.425 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.426 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.427 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.428 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.429 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.430 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.431 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:43.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.431 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:43.489 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:06:43.490 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-01-11T10:06:43.490 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:06:43.490 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-01-11T10:06:43.490 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:06:43.490 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 7.4 M 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:43.491 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:06:43.492 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-01-11T10:06:43.492 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-01-11T10:06:43.492 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:06:43.492 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 38 M 2024-01-11T10:06:43.492 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-01-11T10:06:43.493 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:06:43.530 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-immutable-object-cache-19.0.0-635.g4e3957a 398 kB/s | 147 kB 00:00 2024-01-11T10:06:43.531 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:43.531 INFO:teuthology.orchestra.run.smithi040.stdout:Total 396 kB/s | 147 kB 00:00 2024-01-11T10:06:43.531 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:06:43.539 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:06:43.539 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:06:43.594 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:06:43.594 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:06:43.744 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:06:43.790 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.e 1/1 2024-01-11T10:06:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.e 1/1 2024-01-11T10:06:43.814 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:06:43.814 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-11T10:06:43.814 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:44.154 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 363 kB/s | 195 kB 00:00 2024-01-11T10:06:44.271 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-01-11T10:06:44.305 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-01-11T10:06:44.422 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 156 kB/s | 18 kB 00:00 2024-01-11T10:06:44.505 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 423 kB/s | 35 kB 00:00 2024-01-11T10:06:44.889 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.8 MB/s | 3.0 MB 00:00 2024-01-11T10:06:44.917 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.e 1/1 2024-01-11T10:06:44.918 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:44.918 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:06:44.918 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-immutable-object-cache-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:06:44.918 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:44.918 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:06:44.947 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-01-11T10:06:45.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr 2024-01-11T10:06:45.281 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-63 4.5 MB/s | 7.4 MB 00:01 2024-01-11T10:06:45.490 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-01-11T10:06:45.768 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:14 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:45.898 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.4 MB/s | 5.1 MB 00:00 2024-01-11T10:06:46.075 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:06:46.076 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:06:46.077 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:06:46.078 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:06:46.079 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:06:46.080 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.081 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:46.082 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.083 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.084 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:46.085 INFO:teuthology.orchestra.run.smithi040.stdout:Package ceph-mgr-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:06:46.144 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:06:46.145 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:06:46.146 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:06:46.208 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-dashboard 2024-01-11T10:06:46.265 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.7 MB/s | 4.7 MB 00:01 2024-01-11T10:06:46.815 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:15 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.118 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:06:47.119 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.120 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.121 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:06:47.122 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:06:47.123 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.124 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.125 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:47.126 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.127 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:47.186 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 1.7 M 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 28 k 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 15 k 2024-01-11T10:06:47.187 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-11T10:06:47.188 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-11T10:06:47.188 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:47.188 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:06:47.188 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:06:47.188 INFO:teuthology.orchestra.run.smithi040.stdout:Install 5 Packages 2024-01-11T10:06:47.188 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:47.189 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.0 M 2024-01-11T10:06:47.189 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 8.4 M 2024-01-11T10:06:47.189 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:06:47.451 INFO:teuthology.orchestra.run.smithi040.stdout:(1/5): ceph-prometheus-alerts-19.0.0-635.g4e395 72 kB/s | 15 kB 00:00 2024-01-11T10:06:47.501 INFO:teuthology.orchestra.run.smithi040.stdout:(2/5): ceph-grafana-dashboards-19.0.0-635.g4e39 108 kB/s | 28 kB 00:00 2024-01-11T10:06:47.543 INFO:teuthology.orchestra.run.smithi040.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 335 kB/s | 31 kB 00:00 2024-01-11T10:06:47.601 INFO:teuthology.orchestra.run.smithi040.stdout:(4/5): ceph-mgr-dashboard-19.0.0-635.g4e3957a1. 4.8 MB/s | 1.7 MB 00:00 2024-01-11T10:06:47.685 INFO:teuthology.orchestra.run.smithi040.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-01-11T10:06:47.686 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:47.687 INFO:teuthology.orchestra.run.smithi040.stdout:Total 4.0 MB/s | 2.0 MB 00:00 2024-01-11T10:06:47.687 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:06:47.694 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:06:47.694 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:06:47.754 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:06:47.754 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:06:47.991 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:06:48.067 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-11T10:06:48.106 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-11T10:06:48.137 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-635.g4e3957a1.el9.no 3/5 2024-01-11T10:06:48.387 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-635.g4e3957a1.el9.n 4/5 2024-01-11T10:06:48.421 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 5/5 2024-01-11T10:06:49.014 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 5/5 2024-01-11T10:06:49.014 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-635.g4e3957a1.el9.n 1/5 2024-01-11T10:06:49.014 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 2/5 2024-01-11T10:06:49.014 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-635.g4e3957a1.el9.no 3/5 2024-01-11T10:06:49.014 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-grafana-dashboards-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-prometheus-alerts-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:06:49.339 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-11T10:06:49.340 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-11T10:06:49.340 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:49.340 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:06:49.539 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-11T10:06:50.151 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:19 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:06:50.454 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:06:50.454 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:06:50.454 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:06:50.455 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.456 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:06:50.457 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:06:50.458 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.459 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.460 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.461 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.462 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.463 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.464 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:06:50.525 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:06:50.526 INFO:teuthology.orchestra.run.smithi040.stdout:====================================================================================== 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:====================================================================================== 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 7.4 M 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-01-11T10:06:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout:====================================================================================== 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout:Install 12 Packages 2024-01-11T10:06:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:06:50.529 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 38 M 2024-01-11T10:06:50.529 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 209 M 2024-01-11T10:06:50.529 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:06:51.074 INFO:teuthology.orchestra.run.smithi040.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 882 kB/s | 195 kB 00:00 2024-01-11T10:06:51.116 INFO:teuthology.orchestra.run.smithi040.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-01-11T10:06:51.241 INFO:teuthology.orchestra.run.smithi040.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 196 kB/s | 33 kB 00:00 2024-01-11T10:06:51.292 INFO:teuthology.orchestra.run.smithi040.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-01-11T10:06:51.342 INFO:teuthology.orchestra.run.smithi040.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-01-11T10:06:51.534 INFO:teuthology.orchestra.run.smithi040.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.2 MB/s | 3.0 MB 00:00 2024-01-11T10:06:51.593 INFO:teuthology.orchestra.run.smithi040.stdout:(7/12): python3-devel-3.9.17-2.el9.x86_64.rpm 4.1 MB/s | 245 kB 00:00 2024-01-11T10:06:51.776 INFO:teuthology.orchestra.run.smithi040.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-63 8.0 MB/s | 7.4 MB 00:00 2024-01-11T10:06:51.885 INFO:teuthology.orchestra.run.smithi040.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-01-11T10:06:52.068 INFO:teuthology.orchestra.run.smithi040.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 858 kB/s | 250 kB 00:00 2024-01-11T10:06:52.477 INFO:teuthology.orchestra.run.smithi040.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-01-11T10:06:52.678 INFO:teuthology.orchestra.run.smithi040.stdout:(12/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.6 MB/s | 4.7 MB 00:01 2024-01-11T10:06:52.678 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:52.678 INFO:teuthology.orchestra.run.smithi040.stdout:Total 18 MB/s | 38 MB 00:02 2024-01-11T10:06:52.906 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-01-11T10:06:52.907 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:06:52.907 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.0 MB/s | 38 MB 00:09 2024-01-11T10:06:53.039 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:06:53.064 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:06:53.064 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:06:53.263 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:06:53.289 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:06:53.290 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:06:53.539 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:06:53.539 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:06:53.733 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:06:53.734 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:06:54.469 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:06:54.485 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:06:54.547 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-01-11T10:06:54.560 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-01-11T10:06:54.588 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-01-11T10:06:54.606 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-01-11T10:06:54.695 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-01-11T10:06:54.704 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-01-11T10:06:54.735 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-11T10:06:54.747 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-11T10:06:54.920 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-11T10:06:54.924 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-11T10:06:54.943 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-11T10:06:54.970 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-11T10:06:55.032 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-11T10:06:55.053 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-11T10:06:55.695 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-11T10:06:55.722 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-11T10:06:55.816 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-11T10:06:55.841 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-11T10:06:57.045 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-11T10:06:57.141 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-11T10:06:58.091 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-11T10:06:58.127 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957 12/12 2024-01-11T10:06:58.226 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-11T10:06:58.251 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957 12/12 2024-01-11T10:07:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957 12/12 2024-01-11T10:07:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957 1/12 2024-01-11T10:07:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-11T10:07:00.730 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-01-11T10:07:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-01-11T10:07:00.774 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957 12/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957 1/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 9/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-01-11T10:07:00.775 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-11T10:07:01.241 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-11T10:07:01.242 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-11T10:07:01.242 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-11T10:07:01.242 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:01.242 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:01.263 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-01-11T10:07:01.263 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:01.263 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-diskprediction-local-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:01.264 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:01.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-01-11T10:07:01.650 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-rook 2024-01-11T10:07:02.236 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:31 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:02.258 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:31 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.523 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.524 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.525 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.526 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.527 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.528 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.529 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.530 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.531 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.532 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.571 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:02.571 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:02.571 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:02.572 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.573 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.574 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.575 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:02.576 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.577 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.578 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.579 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.580 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:02.589 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:02.590 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:02.590 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 50 k 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.25.2-1.el9 epel 199 k 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-01-11T10:07:02.591 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-01-11T10:07:02.592 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:02.593 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.2 M 2024-01-11T10:07:02.593 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-01-11T10:07:02.593 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:02.641 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 50 k 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-11T10:07:02.643 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth noarch 1:2.25.2-1.el9 epel 199 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:02.644 INFO:teuthology.orchestra.run.smithi040.stdout:Install 13 Packages 2024-01-11T10:07:02.645 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:02.645 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 2.2 M 2024-01-11T10:07:02.645 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 26 M 2024-01-11T10:07:02.645 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:02.899 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 212 kB/s | 19 kB 00:00 2024-01-11T10:07:02.924 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 222 kB/s | 26 kB 00:00 2024-01-11T10:07:03.008 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-01-11T10:07:03.033 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): ceph-mgr-rook-19.0.0-635.g4e3957a1.el9. 225 kB/s | 50 kB 00:00 2024-01-11T10:07:03.058 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-01-11T10:07:03.084 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-01-11T10:07:03.135 INFO:teuthology.orchestra.run.smithi040.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 195 kB/s | 26 kB 00:00 2024-01-11T10:07:03.142 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 492 kB/s | 54 kB 00:00 2024-01-11T10:07:03.161 INFO:teuthology.orchestra.run.smithi040.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 19 kB 00:00 2024-01-11T10:07:03.186 INFO:teuthology.orchestra.run.smithi040.stdout:(3/13): ceph-mgr-rook-19.0.0-635.g4e3957a1.el9. 278 kB/s | 50 kB 00:00 2024-01-11T10:07:03.270 INFO:teuthology.orchestra.run.smithi040.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-01-11T10:07:03.295 INFO:teuthology.orchestra.run.smithi040.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.4 MB/s | 222 kB 00:00 2024-01-11T10:07:03.320 INFO:teuthology.orchestra.run.smithi040.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-01-11T10:07:03.379 INFO:teuthology.orchestra.run.smithi040.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-01-11T10:07:03.409 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 46 kB/s | 15 kB 00:00 2024-01-11T10:07:03.493 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 74 kB/s | 32 kB 00:00 2024-01-11T10:07:03.562 INFO:teuthology.orchestra.run.smithi040.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-01-11T10:07:03.588 INFO:teuthology.orchestra.run.smithi040.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 110 kB/s | 32 kB 00:00 2024-01-11T10:07:03.693 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-01-11T10:07:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-google-auth-2.25.2-1.el9.noarc 331 kB/s | 199 kB 00:00 2024-01-11T10:07:03.804 INFO:teuthology.orchestra.run.smithi040.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2024-01-11T10:07:03.843 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 597 kB/s | 90 kB 00:00 2024-01-11T10:07:03.871 INFO:teuthology.orchestra.run.smithi040.stdout:(11/13): python3-google-auth-2.25.2-1.el9.noarc 404 kB/s | 199 kB 00:00 2024-01-11T10:07:03.988 INFO:teuthology.orchestra.run.smithi040.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-01-11T10:07:04.010 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-01-11T10:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:04.013 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-01-11T10:07:04.079 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:04.095 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:04.105 INFO:teuthology.orchestra.run.smithi040.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-01-11T10:07:04.105 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:04.106 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-01-11T10:07:04.174 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:04.190 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:04.190 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:04.197 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:04.199 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:04.299 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:04.300 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:04.724 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:04.834 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:04.882 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-11T10:07:04.945 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-11T10:07:05.000 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-11T10:07:05.010 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-11T10:07:05.045 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-11T10:07:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-11T10:07:05.086 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-11T10:07:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-11T10:07:05.174 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-11T10:07:05.179 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-11T10:07:05.217 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-11T10:07:05.290 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.25.2-1.el9.noarch 7/13 2024-01-11T10:07:05.314 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-11T10:07:05.338 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-11T10:07:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-google-auth-1:2.25.2-1.el9.noarch 7/13 2024-01-11T10:07:05.477 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-11T10:07:06.051 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-11T10:07:06.101 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-11T10:07:06.135 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-11T10:07:06.183 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-11T10:07:06.205 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 13/13 2024-01-11T10:07:06.244 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-11T10:07:06.298 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-11T10:07:06.332 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-11T10:07:06.380 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-11T10:07:06.401 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 13/13 2024-01-11T10:07:06.987 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 13/13 2024-01-11T10:07:06.987 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 1/13 2024-01-11T10:07:06.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-11T10:07:06.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-11T10:07:06.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-11T10:07:06.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-11T10:07:06.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-11T10:07:06.988 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-11T10:07:06.989 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-11T10:07:06.989 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-11T10:07:06.989 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.25.2-1.el9.noarch 10/13 2024-01-11T10:07:06.989 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-11T10:07:06.989 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-11T10:07:07.273 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 13/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 1/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-google-auth-1:2.25.2-1.el9.noarch 10/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-11T10:07:07.274 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-11T10:07:07.385 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-11T10:07:07.385 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:07.385 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:07.385 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:07.385 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-11T10:07:07.385 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.25.2-1.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:07.386 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:07.655 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-01-11T10:07:07.687 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-11T10:07:07.687 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:07.687 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-rook-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-google-auth-1:2.25.2-1.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-11T10:07:07.688 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:07.689 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:07.940 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-mgr-cephadm 2024-01-11T10:07:08.260 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:37 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:08.547 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:08.547 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:08.547 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:08.547 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.547 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:08.547 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.548 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.549 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.550 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:08.551 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:08.552 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.553 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.554 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.555 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.556 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.557 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.561 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:37 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:08.614 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:08.616 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:08.616 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:07:08.616 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:08.616 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 133 k 2024-01-11T10:07:08.617 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:07:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-11T10:07:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-01-11T10:07:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-4.el9 appstream 249 k 2024-01-11T10:07:08.617 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-11T10:07:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-11T10:07:08.618 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:08.618 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:08.618 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:08.619 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-01-11T10:07:08.619 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:08.619 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 7.0 M 2024-01-11T10:07:08.619 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-01-11T10:07:08.619 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.873 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.874 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.875 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.876 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:08.877 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.878 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:08.879 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.880 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.881 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:08.943 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 133 k 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-01-11T10:07:08.944 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2 noarch 2.11.3-4.el9 appstream 249 k 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout:Install 6 Packages 2024-01-11T10:07:08.945 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:08.946 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 7.0 M 2024-01-11T10:07:08.946 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 33 M 2024-01-11T10:07:08.946 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:09.427 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 249 kB 00:00 2024-01-11T10:07:09.541 INFO:teuthology.orchestra.run.smithi040.stdout:(1/6): ceph-mgr-cephadm-19.0.0-635.g4e3957a1.el 571 kB/s | 133 kB 00:00 2024-01-11T10:07:09.553 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): ceph-mgr-cephadm-19.0.0-635.g4e3957a1.el 433 kB/s | 133 kB 00:00 2024-01-11T10:07:09.703 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-01-11T10:07:09.728 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 330 kB/s | 58 kB 00:00 2024-01-11T10:07:09.753 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-01-11T10:07:09.875 INFO:teuthology.orchestra.run.smithi040.stdout:(2/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 441 kB/s | 249 kB 00:00 2024-01-11T10:07:09.920 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.9 MB/s | 6.0 MB 00:00 2024-01-11T10:07:09.921 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:09.921 INFO:teuthology.orchestra.run.smithi134.stdout:Total 5.3 MB/s | 7.0 MB 00:01 2024-01-11T10:07:09.990 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:10.001 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:10.001 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:10.100 INFO:teuthology.orchestra.run.smithi040.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 981 kB/s | 548 kB 00:00 2024-01-11T10:07:10.105 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:10.106 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:10.200 INFO:teuthology.orchestra.run.smithi040.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 510 kB/s | 51 kB 00:00 2024-01-11T10:07:10.225 INFO:teuthology.orchestra.run.smithi040.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 164 kB/s | 58 kB 00:00 2024-01-11T10:07:10.301 INFO:teuthology.orchestra.run.smithi040.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 6.0 MB/s | 6.0 MB 00:00 2024-01-11T10:07:10.301 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:10.301 INFO:teuthology.orchestra.run.smithi040.stdout:Total 5.1 MB/s | 7.0 MB 00:01 2024-01-11T10:07:10.374 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:10.385 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:10.385 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:10.422 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:10.494 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:10.494 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:10.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-11T10:07:10.647 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-11T10:07:10.807 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:10.971 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-11T10:07:11.019 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-11T10:07:11.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-11T10:07:11.256 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-11T10:07:11.326 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-11T10:07:11.351 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 6/6 2024-01-11T10:07:11.578 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-11T10:07:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-11T10:07:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-11T10:07:11.773 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 6/6 2024-01-11T10:07:12.157 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 6/6 2024-01-11T10:07:12.158 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/6 2024-01-11T10:07:12.158 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-11T10:07:12.158 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-11T10:07:12.158 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-11T10:07:12.158 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-11T10:07:12.501 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-11T10:07:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-11T10:07:12.502 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:12.502 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 6/6 2024-01-11T10:07:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 1/6 2024-01-11T10:07:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-11T10:07:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-11T10:07:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-11T10:07:12.710 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-11T10:07:12.768 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-01-11T10:07:13.045 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-11T10:07:13.045 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:13.045 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-cephadm-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:13.046 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:13.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-fuse 2024-01-11T10:07:13.379 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:42 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:13.665 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:13.665 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:13.665 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:13.665 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.665 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.666 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.667 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.668 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.669 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:13.670 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:13.671 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.672 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.672 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.672 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.672 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:13.673 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.674 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.675 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:13.730 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 859 k 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-01-11T10:07:13.731 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 939 k 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.1 M 2024-01-11T10:07:13.732 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:13.880 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:42 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:13.955 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 446 kB/s | 80 kB 00:00 2024-01-11T10:07:14.181 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:14.182 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.183 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:14.184 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:14.185 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.186 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.187 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.188 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.189 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.190 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:14.205 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-635.g4e3957a1.el9.x86_6 2.0 MB/s | 859 kB 00:00 2024-01-11T10:07:14.205 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:14.206 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.9 MB/s | 939 kB 00:00 2024-01-11T10:07:14.216 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:14.238 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:14.238 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:14.251 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:14.251 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:14.251 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 859 k 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:Installing dependencies: 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout:Install 2 Packages 2024-01-11T10:07:14.252 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:14.253 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 939 k 2024-01-11T10:07:14.253 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 3.1 M 2024-01-11T10:07:14.253 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:14.316 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:14.317 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:14.511 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:14.586 INFO:teuthology.orchestra.run.smithi040.stdout:(1/2): ceph-fuse-19.0.0-635.g4e3957a1.el9.x86_6 2.9 MB/s | 859 kB 00:00 2024-01-11T10:07:14.648 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-11T10:07:14.721 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 2/2 2024-01-11T10:07:14.786 INFO:teuthology.orchestra.run.smithi040.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 164 kB/s | 80 kB 00:00 2024-01-11T10:07:14.787 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:14.788 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.7 MB/s | 939 kB 00:00 2024-01-11T10:07:14.799 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:14.822 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:14.822 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:14.905 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:14.905 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:15.095 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:15.234 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-11T10:07:15.314 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 2/2 2024-01-11T10:07:15.466 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 2/2 2024-01-11T10:07:15.466 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/2 2024-01-11T10:07:15.842 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-11T10:07:15.842 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:15.842 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:15.842 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-11T10:07:15.842 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:15.842 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:15.988 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 2/2 2024-01-11T10:07:15.988 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/2 2024-01-11T10:07:16.091 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-01-11T10:07:16.329 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-11T10:07:16.329 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:16.329 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:16.329 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-11T10:07:16.329 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:16.329 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:16.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install ceph-volume 2024-01-11T10:07:16.705 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:45 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:16.992 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:16.992 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:16.992 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.993 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:16.994 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.995 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:16.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:16.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:16.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:16.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:17.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:17.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.058 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:17.058 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 262 k 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 262 k 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.2 M 2024-01-11T10:07:17.059 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:17.192 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:46 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:17.309 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-volume-19.0.0-635.g4e3957a1.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-11T10:07:17.309 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:17.310 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-11T10:07:17.310 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:17.317 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:17.317 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:17.357 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:17.357 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:17.494 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:17.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:17.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.501 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:17.502 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.503 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:17.504 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.504 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.504 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.504 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:17.564 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repository Size 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume noarch 2:19.0.0-635.g4e3957a1.el9 ceph-noarch 262 k 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 262 k 2024-01-11T10:07:17.565 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 1.2 M 2024-01-11T10:07:17.566 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:17.573 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:17.644 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:07:17.659 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:07:17.659 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:07:17.659 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-11T10:07:17.660 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:17.808 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-volume-19.0.0-635.g4e3957a1.el9.noarch.rpm 1.1 MB/s | 262 kB 00:00 2024-01-11T10:07:17.808 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:17.809 INFO:teuthology.orchestra.run.smithi040.stdout:Total 1.1 MB/s | 262 kB 00:00 2024-01-11T10:07:17.809 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:17.817 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:17.817 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:17.855 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:17.856 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:18.078 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:18.159 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:07:18.175 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:07:18.175 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:07:18.175 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-11T10:07:18.175 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:18.675 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:07:18.676 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:18.676 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:18.676 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:18.676 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:18.676 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:18.994 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-01-11T10:07:19.285 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 1/1 2024-01-11T10:07:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:19.286 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-volume-2:19.0.0-635.g4e3957a1.el9.noarch 2024-01-11T10:07:19.286 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:19.286 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:19.596 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados-devel 2024-01-11T10:07:19.613 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:48 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:19.903 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:19.903 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:19.904 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.905 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.906 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.907 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.908 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.909 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.910 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.911 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.912 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.913 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:19.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.913 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:19.969 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:19.970 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:19.970 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-11T10:07:19.970 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 127 k 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 457 k 2024-01-11T10:07:19.971 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:20.212 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:49 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:20.313 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-635.g4e3957a1.el9.x86_64. 371 kB/s | 127 kB 00:00 2024-01-11T10:07:20.314 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:20.314 INFO:teuthology.orchestra.run.smithi134.stdout:Total 369 kB/s | 127 kB 00:00 2024-01-11T10:07:20.314 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:20.319 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:20.319 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:20.379 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:20.379 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:20.515 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.516 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.517 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:20.518 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.518 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:20.518 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.518 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:20.518 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.518 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.519 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:20.520 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:20.521 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.522 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.523 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.524 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.525 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:20.532 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:20.585 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 127 k 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:07:20.586 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:20.587 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 127 k 2024-01-11T10:07:20.587 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 457 k 2024-01-11T10:07:20.587 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:20.644 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:20.813 INFO:teuthology.orchestra.run.smithi040.stdout:librados-devel-19.0.0-635.g4e3957a1.el9.x86_64. 562 kB/s | 127 kB 00:00 2024-01-11T10:07:20.813 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:20.813 INFO:teuthology.orchestra.run.smithi040.stdout:Total 559 kB/s | 127 kB 00:00 2024-01-11T10:07:20.814 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:20.819 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:20.819 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:20.881 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:20.881 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:21.030 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:21.102 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:21.138 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:21.433 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:21.433 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:21.433 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:21.433 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:21.433 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:21.433 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:21.606 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:21.696 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-01-11T10:07:21.936 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:21.936 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:21.936 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:21.936 INFO:teuthology.orchestra.run.smithi040.stdout: librados-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:21.937 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:21.937 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:22.189 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs2 2024-01-11T10:07:22.312 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:51 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.602 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.603 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.604 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.605 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:22.606 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:22.607 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.608 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.609 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:22.610 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.611 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.611 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.611 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:22.611 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:22.668 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:22.669 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:22.669 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:22.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-01-11T10:07:22.804 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:51 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:23.107 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:23.108 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.109 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.110 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.111 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:23.112 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.113 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:23.114 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.115 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.116 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.117 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.118 INFO:teuthology.orchestra.run.smithi040.stdout:Package libcephfs2-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:23.178 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:23.179 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:23.179 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:23.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install libcephfs-devel 2024-01-11T10:07:23.366 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:52 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.655 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:23.656 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.657 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:23.658 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.659 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.660 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.661 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:23.662 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.663 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:23.721 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:23.721 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:23.721 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 31 k 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 140 k 2024-01-11T10:07:23.722 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:23.869 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:52 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:23.974 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-635.g4e3957a1.el9.x86_64 124 kB/s | 31 kB 00:00 2024-01-11T10:07:23.974 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:23.974 INFO:teuthology.orchestra.run.smithi134.stdout:Total 123 kB/s | 31 kB 00:00 2024-01-11T10:07:23.974 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:23.978 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:23.978 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:24.001 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:24.001 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:24.085 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:24.173 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:24.174 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:24.175 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.176 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.177 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:24.178 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.179 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:24.180 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.181 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:24.196 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:24.243 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:24.243 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 31 k 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 31 k 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 140 k 2024-01-11T10:07:24.244 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:24.400 INFO:teuthology.orchestra.run.smithi040.stdout:libcephfs-devel-19.0.0-635.g4e3957a1.el9.x86_64 200 kB/s | 31 kB 00:00 2024-01-11T10:07:24.400 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:24.401 INFO:teuthology.orchestra.run.smithi040.stdout:Total 199 kB/s | 31 kB 00:00 2024-01-11T10:07:24.401 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:24.404 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:24.404 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:24.434 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:24.434 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:24.523 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:24.580 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:24.651 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:24.890 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:24.890 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:24.890 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:24.891 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:24.891 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:24.891 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:25.034 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:25.095 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-01-11T10:07:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:25.342 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:25.342 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-devel-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:25.343 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:25.343 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:25.525 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librados2 2024-01-11T10:07:25.695 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:54 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:25.982 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:25.983 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:25.984 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:25.985 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:25.986 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:25.987 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.988 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.989 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.990 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.991 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.992 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.992 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:25.992 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.992 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.992 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.992 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:25.993 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:26.047 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:26.049 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:26.136 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:55 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:26.139 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.439 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.440 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.441 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.442 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:26.443 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.444 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.445 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:26.446 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.447 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:26.448 INFO:teuthology.orchestra.run.smithi040.stdout:Package librados2-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:26.508 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:26.509 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:26.509 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:26.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install librbd1 2024-01-11T10:07:26.737 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:55 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.026 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.027 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.028 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.029 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:27.030 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.031 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.032 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.033 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.034 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:27.091 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:27.092 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:27.092 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:27.180 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-01-11T10:07:27.190 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:56 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:27.491 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.492 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:27.493 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:27.494 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:27.495 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.496 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.497 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.498 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.499 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.500 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:27.501 INFO:teuthology.orchestra.run.smithi040.stdout:Package librbd1-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:27.560 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:27.561 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:27.561 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:27.627 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rados 2024-01-11T10:07:27.778 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:56 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.067 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:28.068 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.069 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:28.070 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.071 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:28.072 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.073 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.074 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.075 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.076 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.077 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:28.133 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:28.134 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:28.134 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:28.223 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-01-11T10:07:28.238 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:57 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.541 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:28.542 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:28.543 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:28.544 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:28.545 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:28.546 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.547 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:28.548 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:28.549 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:28.550 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rados-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:28.611 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:28.613 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:28.613 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:28.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rgw 2024-01-11T10:07:28.824 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:57 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:29.116 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:29.116 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:29.116 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:29.116 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.116 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.117 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:29.118 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:29.119 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:29.120 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.121 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.122 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.123 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.125 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.126 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.126 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.127 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.128 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:29.184 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:29.185 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:29.185 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:29.273 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-01-11T10:07:29.301 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:58 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:29.609 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.610 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.611 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:29.612 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.613 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.614 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.615 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.616 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.617 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.618 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:29.619 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rgw-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:29.680 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:29.681 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:29.681 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:29.751 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-cephfs 2024-01-11T10:07:29.910 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:58 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:30.218 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:30.218 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:30.218 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:30.218 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.218 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:30.218 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:30.219 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:30.220 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.221 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:30.222 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.223 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:30.223 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.223 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:30.223 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.223 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:30.223 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:30.224 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.225 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.226 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.227 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.228 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:30.288 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:30.289 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:30.289 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:30.362 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:02:59 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:30.387 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-01-11T10:07:30.663 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:30.664 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:30.665 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:30.666 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.667 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.668 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.669 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.670 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.671 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:30.672 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-cephfs-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:30.733 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:30.735 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:30.735 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:30.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install python3-rbd 2024-01-11T10:07:30.993 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:00 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:31.284 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:31.285 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:31.286 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.287 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:31.288 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:31.289 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.290 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:31.291 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.292 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.293 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:31.293 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.293 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.293 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.293 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.293 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:31.352 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:31.353 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-01-11T10:07:31.353 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:31.422 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:00 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:31.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-01-11T10:07:31.733 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:31.733 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:31.733 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:31.733 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.733 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:31.734 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.734 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:31.734 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.734 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:31.734 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.734 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:31.735 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.736 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:31.736 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.736 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:31.736 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.736 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:31.736 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.737 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.738 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.739 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:31.740 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:31.741 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.742 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.743 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.743 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:31.743 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.743 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.743 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.743 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.744 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.745 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:31.746 INFO:teuthology.orchestra.run.smithi040.stdout:Package python3-rbd-2:19.0.0-635.g4e3957a1.el9.x86_64 is already installed. 2024-01-11T10:07:31.804 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:31.804 INFO:teuthology.orchestra.run.smithi040.stdout:Nothing to do. 2024-01-11T10:07:31.804 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:31.872 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-fuse 2024-01-11T10:07:32.045 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:01 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.338 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.339 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.340 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.341 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.342 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.343 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.344 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.345 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.346 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.403 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:32.403 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:32.403 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-11T10:07:32.403 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:32.403 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:32.403 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 86 k 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 86 k 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 256 k 2024-01-11T10:07:32.404 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:32.478 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:01 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:32.744 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-635.g4e3957a1.el9.x86_64.rpm 254 kB/s | 86 kB 00:00 2024-01-11T10:07:32.744 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:32.744 INFO:teuthology.orchestra.run.smithi134.stdout:Total 253 kB/s | 86 kB 00:00 2024-01-11T10:07:32.744 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:32.751 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:32.751 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:32.780 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:32.781 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:32.782 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:32.783 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:32.784 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.785 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.786 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.787 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.788 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.789 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:32.809 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:32.809 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:32.849 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 86 k 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:32.850 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:07:32.851 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:32.851 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 86 k 2024-01-11T10:07:32.851 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 256 k 2024-01-11T10:07:32.851 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:32.964 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:33.057 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:33.069 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-fuse-19.0.0-635.g4e3957a1.el9.x86_64.rpm 393 kB/s | 86 kB 00:00 2024-01-11T10:07:33.070 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:33.070 INFO:teuthology.orchestra.run.smithi040.stdout:Total 391 kB/s | 86 kB 00:00 2024-01-11T10:07:33.070 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:33.077 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:33.077 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:33.285 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:33.380 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:33.476 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:33.742 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:33.838 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:33.838 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:33.838 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:33.838 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:33.838 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:34.057 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:34.058 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:34.058 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:34.058 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-fuse-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:34.058 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:34.058 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:34.101 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-01-11T10:07:34.327 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-mirror 2024-01-11T10:07:34.706 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:03 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:34.945 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:03 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.996 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:34.997 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.998 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:34.999 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:35.000 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.001 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:35.002 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.003 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.004 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.005 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:35.005 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.005 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.005 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.005 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.062 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 3.0 M 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.0 M 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 14 M 2024-01-11T10:07:35.063 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.250 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.251 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:35.252 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:35.253 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.254 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.255 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.256 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.257 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.258 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.259 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:35.319 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:35.320 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:35.320 INFO:teuthology.orchestra.run.smithi040.stdout: Package Arch Version Repo Size 2024-01-11T10:07:35.320 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 3.0 M 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 3.0 M 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 14 M 2024-01-11T10:07:35.321 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:35.453 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-635.g4e3957a1.el9.x86_64.rpm 7.8 MB/s | 3.0 MB 00:00 2024-01-11T10:07:35.453 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:35.454 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.8 MB/s | 3.0 MB 00:00 2024-01-11T10:07:35.454 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:35.464 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:35.464 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:35.539 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:35.539 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:35.698 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-mirror-19.0.0-635.g4e3957a1.el9.x86_64.rpm 8.1 MB/s | 3.0 MB 00:00 2024-01-11T10:07:35.699 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:35.699 INFO:teuthology.orchestra.run.smithi040.stdout:Total 8.0 MB/s | 3.0 MB 00:00 2024-01-11T10:07:35.699 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:35.709 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:35.709 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:35.789 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:35.790 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:35.970 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:36.038 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:36.063 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:36.063 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:07:36.063 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-11T10:07:36.063 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-11T10:07:36.063 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-11T10:07:36.063 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:36.247 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:36.307 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:36.331 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:36.332 INFO:teuthology.orchestra.run.smithi040.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-11T10:07:36.332 INFO:teuthology.orchestra.run.smithi040.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-11T10:07:36.332 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-11T10:07:36.332 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-11T10:07:36.332 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:37.220 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:37.220 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:37.220 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:37.220 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:37.220 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:37.220 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:37.471 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-01-11T10:07:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:37.499 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-mirror-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:37.499 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:37.499 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:37.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo yum -y install rbd-nbd 2024-01-11T10:07:38.082 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:07 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:38.366 INFO:teuthology.orchestra.run.smithi040.stdout:Last metadata expiration check: 0:03:07 ago on Thu 11 Jan 2024 10:04:31 AM UTC. 2024-01-11T10:07:38.368 INFO:teuthology.orchestra.run.smithi134.stderr:Modular dependency problems: 2024-01-11T10:07:38.368 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:38.368 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 1: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 2: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 3: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 4: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 5: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 6: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 7: conflicting requests 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.369 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 8: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 9: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 10: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 11: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 12: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 13: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 14: conflicting requests 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.370 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 15: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 16: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 17: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 18: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 19: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 20: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 21: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 22: conflicting requests 2024-01-11T10:07:38.371 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 23: conflicting requests 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 24: conflicting requests 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 25: conflicting requests 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 26: conflicting requests 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 27: conflicting requests 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 28: conflicting requests 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.372 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 29: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 30: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 31: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 32: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 33: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 34: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 35: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 36: conflicting requests 2024-01-11T10:07:38.373 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 37: conflicting requests 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 39: conflicting requests 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.374 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:38.375 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.376 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.377 INFO:teuthology.orchestra.run.smithi134.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.436 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-01-11T10:07:38.436 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:38.436 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-01-11T10:07:38.436 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:38.436 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-01-11T10:07:38.436 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 169 k 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 169 k 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 537 k 2024-01-11T10:07:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr:Modular dependency problems: 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 1: conflicting requests 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240105104119:20125149.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 2: conflicting requests 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 3: conflicting requests 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.679 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 4: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 5: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820231031173453:b754926a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 6: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820230817140043:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 7: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module idm:client:820231122122737:49cc9d1b.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 8: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 9: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 10: conflicting requests 2024-01-11T10:07:38.680 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240105093033:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 11: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 12: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 13: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 14: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 15: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 16: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 17: conflicting requests 2024-01-11T10:07:38.681 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 18: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 19: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 20: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 21: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 22: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 23: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820231201124232:182f7c73.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 24: conflicting requests 2024-01-11T10:07:38.682 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 25: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 26: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820231018065245:17377f89.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 27: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 28: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 29: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 30: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240108205145:b09eea91.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 31: conflicting requests 2024-01-11T10:07:38.683 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 32: conflicting requests 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module squid:4:820230817231530:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 33: conflicting requests 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 34: conflicting requests 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 35: conflicting requests 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 36: conflicting requests 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240104232631:9edba152.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 37: conflicting requests 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.684 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 38: cannot install the best candidate for the job 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 39: conflicting requests 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 40: cannot install the best candidate for the job 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.685 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 41: cannot install the best candidate for the job 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 42: cannot install the best candidate for the job 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.686 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 43: cannot install the best candidate for the job 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 44: cannot install the best candidate for the job 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: Problem 45: cannot install the best candidate for the job 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.687 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.688 INFO:teuthology.orchestra.run.smithi040.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-01-11T10:07:38.750 INFO:teuthology.orchestra.run.smithi040.stdout:Dependencies resolved. 2024-01-11T10:07:38.750 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: Package Architecture Version Repository Size 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Installing: 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd x86_64 2:19.0.0-635.g4e3957a1.el9 ceph 169 k 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction Summary 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:================================================================================ 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Install 1 Package 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Total download size: 169 k 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Installed size: 537 k 2024-01-11T10:07:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Downloading Packages: 2024-01-11T10:07:38.789 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-635.g4e3957a1.el9.x86_64.rpm 480 kB/s | 169 kB 00:00 2024-01-11T10:07:38.790 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:38.790 INFO:teuthology.orchestra.run.smithi134.stdout:Total 478 kB/s | 169 kB 00:00 2024-01-11T10:07:38.790 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-01-11T10:07:38.797 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-01-11T10:07:38.798 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-01-11T10:07:38.857 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-01-11T10:07:38.857 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-01-11T10:07:38.985 INFO:teuthology.orchestra.run.smithi040.stdout:rbd-nbd-19.0.0-635.g4e3957a1.el9.x86_64.rpm 727 kB/s | 169 kB 00:00 2024-01-11T10:07:38.986 INFO:teuthology.orchestra.run.smithi040.stdout:-------------------------------------------------------------------------------- 2024-01-11T10:07:38.986 INFO:teuthology.orchestra.run.smithi040.stdout:Total 719 kB/s | 169 kB 00:00 2024-01-11T10:07:38.986 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction check 2024-01-11T10:07:39.001 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction check succeeded. 2024-01-11T10:07:39.001 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction test 2024-01-11T10:07:39.014 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-01-11T10:07:39.070 INFO:teuthology.orchestra.run.smithi040.stdout:Transaction test succeeded. 2024-01-11T10:07:39.071 INFO:teuthology.orchestra.run.smithi040.stdout:Running transaction 2024-01-11T10:07:39.111 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:39.225 INFO:teuthology.orchestra.run.smithi040.stdout: Preparing : 1/1 2024-01-11T10:07:39.335 INFO:teuthology.orchestra.run.smithi040.stdout: Installing : rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:39.552 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:39.786 INFO:teuthology.orchestra.run.smithi040.stdout: Running scriptlet: rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:39.879 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:39.880 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-01-11T10:07:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:39.880 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:39.880 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-01-11T10:07:40.148 DEBUG:teuthology.parallel:result is None 2024-01-11T10:07:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: Verifying : rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 1/1 2024-01-11T10:07:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:40.149 INFO:teuthology.orchestra.run.smithi040.stdout:Installed: 2024-01-11T10:07:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: rbd-nbd-2:19.0.0-635.g4e3957a1.el9.x86_64 2024-01-11T10:07:40.149 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:40.149 INFO:teuthology.orchestra.run.smithi040.stdout:Complete! 2024-01-11T10:07:40.429 DEBUG:teuthology.parallel:result is None 2024-01-11T10:07:40.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e3957a13c3b8938fe41053b13fb80161e1916b7 2024-01-11T10:07:40.688 DEBUG:teuthology.orchestra.run.smithi040:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-11T10:07:40.709 INFO:teuthology.orchestra.run.smithi040.stdout:19.0.0-635.g4e3957a1.el9 2024-01-11T10:07:40.709 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-635.g4e3957a1.el9 2024-01-11T10:07:40.709 INFO:teuthology.task.install:The correct ceph version 19.0.0-635.g4e3957a1 is installed. 2024-01-11T10:07:40.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=4e3957a13c3b8938fe41053b13fb80161e1916b7 2024-01-11T10:07:40.983 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-11T10:07:41.005 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-635.g4e3957a1.el9 2024-01-11T10:07:41.005 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-635.g4e3957a1.el9 2024-01-11T10:07:41.005 INFO:teuthology.task.install:The correct ceph version 19.0.0-635.g4e3957a1 is installed. 2024-01-11T10:07:41.007 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-11T10:07:41.007 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:41.007 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T10:07:41.040 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:41.040 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-11T10:07:41.075 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-11T10:07:41.076 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:41.076 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T10:07:41.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T10:07:41.180 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:41.181 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-01-11T10:07:41.209 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-11T10:07:41.278 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-11T10:07:41.279 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:41.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T10:07:41.310 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T10:07:41.380 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:41.380 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-11T10:07:41.409 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-11T10:07:41.477 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-11T10:07:41.478 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:41.478 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T10:07:41.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T10:07:41.579 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:41.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-01-11T10:07:41.607 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-11T10:07:41.675 INFO:teuthology.run_tasks:Running task ceph... 2024-01-11T10:07:41.771 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-11T10:07:41.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2024-01-11T10:07:41.774 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-01-11T10:07:41.802 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-11T10:07:41.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-11T10:07:41.846 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-11T10:07:41.874 INFO:tasks.ceph:Creating extra log directories... 2024-01-11T10:07:41.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-11T10:07:41.917 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-11T10:07:41.956 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-11T10:07:41.956 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-11T10:07:41.957 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429', 'branch': 'wip-matanb-crimson-multicore-msgr', 'description': 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7513429', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-multicore-msgr'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'f14f60afd567d61490bfe92caba04c0ffb5b09d2'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 109, 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'f14f60afd567d61490bfe92caba04c0ffb5b09d2', 'targets': {'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9FCjzta3j/kTl2AOjmnxEItzktKfjKINZVSKfOMgwX2Ba7WsHwXv+m5Q1BXECZwcz6kkuR8ASgaWKtlP9UxeI=', 'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEp1dt+7AFCwJg78Y81xcft8Bn5sEh6vOHvj2vbxh5jm9Qm1g1pyyMHmf1Suc/91Z1i3t0pR8S2AcXCmiT4zS4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'full_sequential': [{'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}, {'radosbench': {'clients': ['client.0'], 'time': 90}}]}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-11_09:46:59', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-11T10:07:41.957 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-11T10:07:41.989 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-11T10:07:42.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2024-01-11T10:07:42.047 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-01-11T10:07:42.076 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:42.076 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-01-11T10:07:42.134 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-11T10:07:42.134 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-01-11T10:07:42.191 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-11T10:07:42.191 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:42.191 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-01-11T10:07:42.192 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:42.192 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:42.192 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-01-11 10:07:36.582206274 +0000 2024-01-11T10:07:42.192 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-01-11 10:03:36.552421560 +0000 2024-01-11T10:07:42.192 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-01-11 10:03:36.552421560 +0000 2024-01-11T10:07:42.192 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-01-11 10:03:36.552421560 +0000 2024-01-11T10:07:42.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-11T10:07:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-01-11T10:07:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-01-11T10:07:42.261 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000261511 s, 2.0 MB/s 2024-01-11T10:07:42.263 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-11T10:07:42.321 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-01-11T10:07:42.378 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-11T10:07:42.378 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:42.378 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-11T10:07:42.379 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:42.379 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:42.379 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-01-11 10:07:36.582206274 +0000 2024-01-11T10:07:42.379 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-01-11 10:03:36.901417074 +0000 2024-01-11T10:07:42.379 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-01-11 10:03:36.901417074 +0000 2024-01-11T10:07:42.379 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-01-11 10:03:36.901417074 +0000 2024-01-11T10:07:42.379 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-11T10:07:42.449 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-01-11T10:07:42.449 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-01-11T10:07:42.449 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000198134 s, 2.6 MB/s 2024-01-11T10:07:42.450 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-11T10:07:42.509 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-01-11 10:07:36.583206261 +0000 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-01-11 10:03:37.246412640 +0000 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-01-11 10:03:37.246412640 +0000 2024-01-11T10:07:42.566 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-01-11 10:03:37.246412640 +0000 2024-01-11T10:07:42.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-11T10:07:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-01-11T10:07:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-01-11T10:07:42.635 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000176743 s, 2.9 MB/s 2024-01-11T10:07:42.636 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-11T10:07:42.694 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-01-11 10:07:36.583206261 +0000 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-01-11 10:03:37.591408206 +0000 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-01-11 10:03:37.591408206 +0000 2024-01-11T10:07:42.751 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: 2024-01-11 10:03:37.591408206 +0000 2024-01-11T10:07:42.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-11T10:07:42.819 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-01-11T10:07:42.819 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-01-11T10:07:42.819 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000240096 s, 2.1 MB/s 2024-01-11T10:07:42.821 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-11T10:07:42.878 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-01-11T10:07:42.878 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:42.878 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-01-11T10:07:42.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-11T10:07:42.897 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-11 10:07:36.337312552 +0000 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-11 10:03:22.822815687 +0000 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-11 10:03:22.822815687 +0000 2024-01-11T10:07:42.955 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-11 10:03:22.822815687 +0000 2024-01-11T10:07:42.956 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-11T10:07:43.024 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-11T10:07:43.024 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-11T10:07:43.024 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000217162 s, 2.4 MB/s 2024-01-11T10:07:43.025 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-11T10:07:43.084 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-01-11T10:07:43.141 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-11T10:07:43.141 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:43.141 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-01-11T10:07:43.141 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:43.142 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:43.142 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-11 10:07:36.337312552 +0000 2024-01-11T10:07:43.142 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-11 10:03:23.353803307 +0000 2024-01-11T10:07:43.142 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-11 10:03:23.353803307 +0000 2024-01-11T10:07:43.142 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-11 10:03:23.353803307 +0000 2024-01-11T10:07:43.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-11T10:07:43.209 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-11T10:07:43.209 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-11T10:07:43.209 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000195784 s, 2.6 MB/s 2024-01-11T10:07:43.210 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-11T10:07:43.267 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-11 10:07:36.338312529 +0000 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-11 10:03:23.689795472 +0000 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-11 10:03:23.689795472 +0000 2024-01-11T10:07:43.323 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-11 10:03:23.689795472 +0000 2024-01-11T10:07:43.324 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-11T10:07:43.389 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-11T10:07:43.390 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-11T10:07:43.390 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00016528 s, 3.1 MB/s 2024-01-11T10:07:43.391 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-11T10:07:43.447 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-01-11T10:07:43.503 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-11T10:07:43.503 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-11T10:07:43.504 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-01-11T10:07:43.504 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-11T10:07:43.504 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-01-11T10:07:43.504 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-01-11 10:07:36.338312529 +0000 2024-01-11T10:07:43.504 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-01-11 10:03:24.039787312 +0000 2024-01-11T10:07:43.504 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-01-11 10:03:24.039787312 +0000 2024-01-11T10:07:43.505 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-01-11 10:03:24.039787312 +0000 2024-01-11T10:07:43.505 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-11T10:07:43.569 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-01-11T10:07:43.569 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-01-11T10:07:43.569 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000178559 s, 2.9 MB/s 2024-01-11T10:07:43.570 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-11T10:07:43.626 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-01-11T10:07:43.626 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-01-11T10:07:43.626 INFO:tasks.ceph:Generating config... 2024-01-11T10:07:43.627 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-11T10:07:43.627 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-11T10:07:43.627 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-01-11T10:07:43.627 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] err to stderr = True 2024-01-11T10:07:43.628 INFO:tasks.ceph:[osd] log flush on exit = True 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] log to file = False 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] log to stderr = True 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd delete sleep = 1 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd max backfills = 3 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-01-11T10:07:43.629 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-11T10:07:43.630 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2024-01-11T10:07:43.630 INFO:tasks.ceph:[client.0] debug ms = 1 2024-01-11T10:07:43.630 INFO:tasks.ceph:[client.0] debug objecter = 20 2024-01-11T10:07:43.630 INFO:tasks.ceph:[client.0] debug rados = 20 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] ms client mode = crc 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] ms service mode = crc 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-01-11T10:07:43.630 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2024-01-11T10:07:43.630 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-11T10:07:43.630 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-11T10:07:43.630 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-01-11T10:07:43.631 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-01-11T10:07:43.632 INFO:tasks.ceph:Setting up mon.a... 2024-01-11T10:07:43.632 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-11T10:07:43.692 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.keyring 2024-01-11T10:07:43.695 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-11T10:07:43.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T10:07:43.777 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.40')] 2024-01-11T10:07:43.777 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.40', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}} 2024-01-11T10:07:43.778 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:43.778 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-11T10:07:43.832 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.40 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:43.917 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:07:43.932+0000 7f2380064e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:07:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:07:43.932+0000 7f2380064e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:07:43.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:07:43.933+0000 7f2380064e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:07:43.918 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:43.918 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: generated fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:07:43.918 INFO:teuthology.orchestra.run.smithi040.stdout:setting min_mon_release = quincy 2024-01-11T10:07:43.918 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 0 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:last_changed 2024-01-11T10:07:43.933682+0000 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-01-11T10:07:43.933682+0000 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:min_mon_release 17 (quincy) 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:election_strategy: 1 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:0: [v2:172.21.15.40:3300/0,v1:172.21.15.40:6789/0] mon.a 2024-01-11T10:07:43.919 INFO:teuthology.orchestra.run.smithi040.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-11T10:07:43.920 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-11T10:07:43.934 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40... 2024-01-11T10:07:43.935 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-11T10:07:43.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2024-01-11T10:07:44.025 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.026 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2024-01-11T10:07:44.027 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2024-01-11T10:07:44.028 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = 172.21.15.40 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: ms client mode = crc 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: ms cluster mode = crc 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon client mode = crc 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon cluster mode = crc 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: ms mon service mode = crc 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: ms service mode = crc 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_min_size = 2 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_size = 3 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2024-01-11T10:07:44.029 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 120 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug trim objects = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2024-01-11T10:07:44.030 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2024-01-11T10:07:44.031 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression algorithm = snappy 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression mode = aggressive 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore rocksdb cf = False 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: crimson seastar smp = 3 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 20 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 20 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: err to stderr = True 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: log flush on exit = True 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: log to file = False 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: log to stderr = True 2024-01-11T10:07:44.032 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd blocked scrub grace period = 3600 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug reject backfill probability = 0.3 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd delete sleep = 1 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd max backfills = 3 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd max markdown count = 1000 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub min interval = 60 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: osd snap trim sleep = 2 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2024-01-11T10:07:44.033 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2024-01-11T10:07:44.034 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2024-01-11T10:07:44.035 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: mon min osdmap epochs = 50 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune interval = 2 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune min = 15 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune txsize = 2 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default crimson = True 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: paxos service trim min = 10 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-11T10:07:44.036 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi040.stdout: debug objecter = 20 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi040.stdout: debug rados = 20 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-01-11T10:07:44.037 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.038 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-01-11T10:07:44.039 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = 172.21.15.40 2024-01-11T10:07:44.040 INFO:teuthology.orchestra.run.smithi134.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: ms client mode = crc 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: ms cluster mode = crc 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: ms mon client mode = crc 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: ms mon cluster mode = crc 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: ms mon service mode = crc 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: ms service mode = crc 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_min_size = 2 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_size = 3 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.041 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 120 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.042 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression algorithm = snappy 2024-01-11T10:07:44.043 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore compression mode = aggressive 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore rocksdb cf = False 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: crimson seastar smp = 3 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 20 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 20 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 10 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: err to stderr = True 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: log flush on exit = True 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: log to file = False 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: log to stderr = True 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-01-11T10:07:44.044 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd blocked scrub grace period = 3600 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug reject backfill probability = 0.3 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd delete sleep = 1 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd max backfills = 3 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd max markdown count = 1000 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub min interval = 60 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd shutdown pgref assert = True 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: osd snap trim sleep = 2 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-11T10:07:44.045 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-01-11T10:07:44.046 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.047 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: mon min osdmap epochs = 50 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune interval = 2 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune min = 15 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune txsize = 2 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default crimson = True 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: paxos service trim min = 10 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-01-11T10:07:44.048 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-01-11T10:07:44.049 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-01-11T10:07:44.049 INFO:teuthology.orchestra.run.smithi134.stdout: debug objecter = 20 2024-01-11T10:07:44.049 INFO:teuthology.orchestra.run.smithi134.stdout: debug rados = 20 2024-01-11T10:07:44.050 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-11T10:07:44.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-11T10:07:44.151 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-11T10:07:44.151 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:44.151 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-11T10:07:44.165 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:44.165 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-11T10:07:44.220 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T10:07:44.220 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:44.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-11T10:07:44.221 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T10:07:44.299 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:07:44.299 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:44.353 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T10:07:44.353 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:44.353 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-11T10:07:44.354 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-11T10:07:44.394 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:07:44.394 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:44.450 INFO:tasks.ceph:Setting up mon nodes... 2024-01-11T10:07:44.450 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-11T10:07:44.450 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-11T10:07:44.566 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-11T10:07:44.569 INFO:tasks.ceph:Setting up mds nodes... 2024-01-11T10:07:44.569 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-11T10:07:44.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-11T10:07:44.617 INFO:teuthology.orchestra.run.smithi040.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-11T10:07:44.633 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-11T10:07:44.681 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-11T10:07:44.696 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-11T10:07:44.697 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2024-01-11T10:07:44.697 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-11T10:07:44.723 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-01-11T10:07:44.723 INFO:tasks.ceph:role: osd.0 2024-01-11T10:07:44.724 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T10:07:44.724 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T10:07:44.798 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T10:07:45.311 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-01-11T10:07:45.313 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-01-11T10:07:45.313 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-11T10:07:45.445 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-11T10:07:45.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-11T10:07:45.547 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2024-01-11T10:07:45.547 INFO:tasks.ceph:role: osd.1 2024-01-11T10:07:45.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T10:07:45.547 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-11T10:07:45.622 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T10:07:45.623 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T10:07:46.071 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-01-11T10:07:46.073 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-01-11T10:07:46.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-11T10:07:46.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-11T10:07:46.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:07:46.733 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,749 seastar - Reactor backend: linux-aio 2024-01-11T10:07:46.749 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,765 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:07:46.749 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,765 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:07:46.750 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,765 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:07:46.750 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,765 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:07:46.751 INFO:teuthology.orchestra.run.smithi040.stderr:==27783==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:07:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:==27783==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd15dc6000; bottom 0x7fe1e711a000; size: 0x001b2ecac000 (116749156352) 2024-01-11T10:07:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2024-01-11T10:07:46.758 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:07:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e7607e2 0x2e761b90 0x2e7630c2 0x2e763446 0x2e763590 0x2e7639e6 0x54daf 0x240d97fc 0x240df3f9 0x23ea64aa 0x23f289c9 0x25df0a54 0x25df8588 0x25dfaed5 0x25dfc09a 0x23a054a3 0x23a2240a 0x23a96a7b 0x2e6d319b 0x2e7363cc 0x2e9dc4f5 0x2e9de33f 0x2e38f5e6 0x2e3914ef 0x1f8a3cf3 0x1f8aa79b 0x1f681ba4 0x3feaf 0x3ff5f 0x1f241b54 2024-01-11T10:07:46.784 INFO:teuthology.orchestra.run.smithi040.stderr:kernel callstack: 2024-01-11T10:07:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,812 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2024-01-11T10:07:46.796 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:46,812 [shard 0] osd - get_early_config: set --smp 3 2024-01-11T10:07:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:47.296 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-01-11T10:07:47.297 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-01-11T10:07:47.297 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-01-11T10:07:47.297 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:47.297 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:47.349 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:07:47.350 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:47,366 seastar - Reactor backend: linux-aio 2024-01-11T10:07:47.378 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:47,393 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:07:47.378 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:47,393 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:07:47.378 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:47,393 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:07:47.385 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:47,400 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:07:47.387 INFO:teuthology.orchestra.run.smithi040.stderr:==27781==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:07:47.393 INFO:teuthology.orchestra.run.smithi040.stderr:==27781==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd15dc6000; bottom 0x7fe1e4f03000; size: 0x001b30ec3000 (116784902144) 2024-01-11T10:07:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2024-01-11T10:07:47.394 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:07:51.422 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:51.422 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-01-11T10:07:51.422 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-01-11T10:07:51.422 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-01-11T10:07:51.422 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:51.423 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:51.423 INFO:teuthology.orchestra.run.smithi040.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:07:51.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-11T10:07:51.489 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:07:52.016 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,032 seastar - Reactor backend: linux-aio 2024-01-11T10:07:52.032 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,047 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:07:52.032 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,048 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:07:52.032 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,048 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:07:52.033 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,048 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:07:52.034 INFO:teuthology.orchestra.run.smithi040.stderr:==27965==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:07:52.040 INFO:teuthology.orchestra.run.smithi040.stderr:==27965==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffeaef0b000; bottom 0x7fd3b6dc3000; size: 0x002af8148000 (184550719488) 2024-01-11T10:07:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2024-01-11T10:07:52.041 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:07:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,108 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2024-01-11T10:07:52.093 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,108 [shard 0] osd - get_early_config: set --smp 3 2024-01-11T10:07:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:52.590 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-01-11T10:07:52.590 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-01-11T10:07:52.590 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-01-11T10:07:52.591 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:52.591 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:52.643 INFO:teuthology.orchestra.run.smithi040.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:07:52.644 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,659 seastar - Reactor backend: linux-aio 2024-01-11T10:07:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,677 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:07:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,677 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:07:52.662 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,677 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:07:52.669 INFO:teuthology.orchestra.run.smithi040.stderr:INFO 2024-01-11 10:07:52,685 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:07:52.672 INFO:teuthology.orchestra.run.smithi040.stderr:==27963==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:07:52.678 INFO:teuthology.orchestra.run.smithi040.stderr:==27963==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffeaef0b000; bottom 0x7fd3b4bac000; size: 0x002afa35f000 (184586465280) 2024-01-11T10:07:52.678 INFO:teuthology.orchestra.run.smithi040.stderr:False positive error reports may follow 2024-01-11T10:07:52.678 INFO:teuthology.orchestra.run.smithi040.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:07:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:Suppressions used: 2024-01-11T10:07:56.737 INFO:teuthology.orchestra.run.smithi040.stderr: count bytes template 2024-01-11T10:07:56.737 INFO:teuthology.orchestra.run.smithi040.stderr: 1 8 MallocExtension::Register 2024-01-11T10:07:56.737 INFO:teuthology.orchestra.run.smithi040.stderr:----------------------------------------------------- 2024-01-11T10:07:56.738 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:07:56.738 INFO:teuthology.orchestra.run.smithi040.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:07:56.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-11T10:07:56.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-11T10:07:56.837 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-01-11T10:07:56.837 INFO:tasks.ceph:role: osd.2 2024-01-11T10:07:56.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T10:07:56.837 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-11T10:07:56.916 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T10:07:56.916 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T10:07:56.916 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T10:07:56.917 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T10:07:57.412 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-01-11T10:07:57.414 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-01-11T10:07:57.414 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-01-11T10:07:57.554 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-11T10:07:57.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-11T10:07:57.664 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2024-01-11T10:07:57.664 INFO:tasks.ceph:role: osd.3 2024-01-11T10:07:57.664 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T10:07:57.665 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-11T10:07:57.741 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-11T10:07:57.742 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-11T10:07:57.742 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-11T10:07:58.182 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-01-11T10:07:58.183 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-01-11T10:07:58.184 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2024-01-11T10:07:58.272 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-11T10:07:58.302 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:07:58.859 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:07:58.860 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,875 seastar - Reactor backend: linux-aio 2024-01-11T10:07:58.884 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,899 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:07:58.884 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,899 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:07:58.884 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,899 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:07:58.884 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,899 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:07:58.886 INFO:teuthology.orchestra.run.smithi134.stderr:==27587==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:07:58.892 INFO:teuthology.orchestra.run.smithi134.stderr:==27587==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc429ce000; bottom 0x7f914df43000; size: 0x006af4a8b000 (459371229184) 2024-01-11T10:07:58.892 INFO:teuthology.orchestra.run.smithi134.stderr:False positive error reports may follow 2024-01-11T10:07:58.892 INFO:teuthology.orchestra.run.smithi134.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:07:58.930 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,945 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2024-01-11T10:07:58.930 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:58,945 [shard 0] osd - get_early_config: set --smp 3 2024-01-11T10:07:59.422 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:07:59.422 INFO:teuthology.orchestra.run.smithi134.stderr:Suppressions used: 2024-01-11T10:07:59.422 INFO:teuthology.orchestra.run.smithi134.stderr: count bytes template 2024-01-11T10:07:59.422 INFO:teuthology.orchestra.run.smithi134.stderr: 1 8 MallocExtension::Register 2024-01-11T10:07:59.422 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:07:59.422 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:07:59.471 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:07:59.471 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:59,487 seastar - Reactor backend: linux-aio 2024-01-11T10:07:59.489 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:59,505 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:07:59.489 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:59,505 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:07:59.489 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:59,505 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:07:59.497 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:07:59,512 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:07:59.499 INFO:teuthology.orchestra.run.smithi134.stderr:==27583==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:07:59.505 INFO:teuthology.orchestra.run.smithi134.stderr:==27583==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc429ce000; bottom 0x7f914bd2c000; size: 0x006af6ca2000 (459406974976) 2024-01-11T10:07:59.505 INFO:teuthology.orchestra.run.smithi134.stderr:False positive error reports may follow 2024-01-11T10:07:59.505 INFO:teuthology.orchestra.run.smithi134.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:08:03.497 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:08:03.498 INFO:teuthology.orchestra.run.smithi134.stderr:Suppressions used: 2024-01-11T10:08:03.498 INFO:teuthology.orchestra.run.smithi134.stderr: count bytes template 2024-01-11T10:08:03.498 INFO:teuthology.orchestra.run.smithi134.stderr: 1 8 MallocExtension::Register 2024-01-11T10:08:03.498 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:08:03.498 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:08:03.498 INFO:teuthology.orchestra.run.smithi134.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:08:03.535 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-11T10:08:03.563 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:08:04.082 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:08:04.083 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,098 seastar - Reactor backend: linux-aio 2024-01-11T10:08:04.093 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,108 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:08:04.093 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,108 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:08:04.093 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,108 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:08:04.093 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,108 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:08:04.095 INFO:teuthology.orchestra.run.smithi134.stderr:==27770==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:08:04.101 INFO:teuthology.orchestra.run.smithi134.stderr:==27770==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc80d17000; bottom 0x7f853a3c8000; size: 0x00774694f000 (512285274112) 2024-01-11T10:08:04.101 INFO:teuthology.orchestra.run.smithi134.stderr:False positive error reports may follow 2024-01-11T10:08:04.101 INFO:teuthology.orchestra.run.smithi134.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:08:04.136 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,152 [shard 0] osd - bypassing the config fetch due to --no-mon-config 2024-01-11T10:08:04.137 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,152 [shard 0] osd - get_early_config: set --smp 3 2024-01-11T10:08:04.604 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:08:04.604 INFO:teuthology.orchestra.run.smithi134.stderr:Suppressions used: 2024-01-11T10:08:04.604 INFO:teuthology.orchestra.run.smithi134.stderr: count bytes template 2024-01-11T10:08:04.604 INFO:teuthology.orchestra.run.smithi134.stderr: 1 8 MallocExtension::Register 2024-01-11T10:08:04.604 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:08:04.604 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:08:04.662 INFO:teuthology.orchestra.run.smithi134.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-11T10:08:04.662 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,678 seastar - Reactor backend: linux-aio 2024-01-11T10:08:04.696 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,712 [shard 2] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-11T10:08:04.697 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,712 [shard 2] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-11T10:08:04.697 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,712 [shard 2] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-11T10:08:04.697 INFO:teuthology.orchestra.run.smithi134.stderr:INFO 2024-01-11 10:08:04,712 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-11T10:08:04.699 INFO:teuthology.orchestra.run.smithi134.stderr:==27768==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-11T10:08:04.704 INFO:teuthology.orchestra.run.smithi134.stderr:==27768==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc80d17000; bottom 0x7f85381b1000; size: 0x007748b66000 (512321019904) 2024-01-11T10:08:04.705 INFO:teuthology.orchestra.run.smithi134.stderr:False positive error reports may follow 2024-01-11T10:08:04.705 INFO:teuthology.orchestra.run.smithi134.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stderr:Suppressions used: 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stderr: count bytes template 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stderr: 1 8 MallocExtension::Register 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stderr:----------------------------------------------------- 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T10:08:08.709 INFO:teuthology.orchestra.run.smithi134.stdout:created object store /var/lib/ceph/osd/ceph-3 for osd.3 fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:08:08.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-11T10:08:08.774 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-11T10:08:08.774 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:08:08.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-11T10:08:08.805 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:08:08.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-11T10:08:08.875 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:08:08.875 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-11T10:08:08.902 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:08:08.902 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-11T10:08:08.969 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:08:08.969 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-11T10:08:09.036 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:08:09.037 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-11T10:08:09.053 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:08:09.053 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-11T10:08:09.091 INFO:tasks.ceph:Adding keys to all mons... 2024-01-11T10:08:09.091 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDzvZ9la9h2GxAAegJFd1o0dHtc8NJ0wO+GJg== 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD4vZ9lRDNbLBAAmS6KgVEMrrucLf6QYypynw== 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDwvZ9lT+KuIhAAai8FpPio0ln776xc1tdz9g== 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD/vZ9lhaLrIRAA8hK2iE/T3LPK7bm77LIvsg== 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAEvp9lc3vYLRAAcDiKsY6cV+d4O2zKD2i7oA== 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-01-11T10:08:09.122 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDwvZ9lpZW9JRAADuedOzUkUnoFx60u6irZUw== 2024-01-11T10:08:09.123 INFO:teuthology.orchestra.run.smithi040.stdout:[client.1] 2024-01-11T10:08:09.123 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDwvZ9lM7KIKRAASlFWcY3GHT6N6QvAKtWwdg== 2024-01-11T10:08:09.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T10:08:09.227 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T10:08:09.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-11T10:08:09.343 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T10:08:09.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-11T10:08:09.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T10:08:09.517 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-11T10:08:09.577 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-11T10:08:09.578 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-11T10:08:09.604 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-11T10:08:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:09.714+0000 7fc5a106e9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.700 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:09.714+0000 7fc5a106e9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.713 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-11T10:08:09.739 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-11T10:08:09.793 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-11T10:08:09.794 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-11T10:08:09.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-11T10:08:09.836 INFO:tasks.ceph.mon.a:Started 2024-01-11T10:08:09.836 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-11T10:08:09.836 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-11T10:08:09.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-11T10:08:09.838 INFO:tasks.ceph.mgr.x:Started 2024-01-11T10:08:09.839 INFO:tasks.ceph:Setting crush tunables to default 2024-01-11T10:08:09.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd crush tunables default 2024-01-11T10:08:09.914 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-01-11T10:08:09.928+0000 7f45fee2a9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.915 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-01-11T10:08:09.928+0000 7f45fee2a9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.915 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-01-11T10:08:09.929+0000 7f45fee2a9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.925 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:09.939+0000 7f94e10f5f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.946 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:09.955+0000 7efe0e19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.964 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:09.979+0000 7f94e10f5f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.964 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:09.979+0000 7f94e10f5f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:09.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:09.978+0000 7efe0e19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:10.037 INFO:teuthology.orchestra.run.smithi040.stderr:adjusted tunables profile to default 2024-01-11T10:08:10.048 INFO:tasks.ceph:check_enable_crimson: True 2024-01-11T10:08:10.048 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-01-11T10:08:10.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-01-11T10:08:10.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:10.162+0000 7f95c2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:10.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:10.170+0000 7f95c2c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:10.238 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-01-11T10:08:10.252+0000 7f45f6df1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-01-11T10:08:10.238 INFO:tasks.ceph.mon.a.smithi040.stderr:Please be aware that this feature is experimental, untested, 2024-01-11T10:08:10.238 INFO:tasks.ceph.mon.a.smithi040.stderr:unsupported, and may result in data corruption, data loss, 2024-01-11T10:08:10.238 INFO:tasks.ceph.mon.a.smithi040.stderr:and/or irreparable damage to your cluster. Do not use 2024-01-11T10:08:10.238 INFO:tasks.ceph.mon.a.smithi040.stderr:feature with important data. 2024-01-11T10:08:10.238 INFO:tasks.ceph.mon.a.smithi040.stderr: 2024-01-11T10:08:10.413 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:10.428+0000 7f94e10f5f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-11T10:08:10.464 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:10.478+0000 7f94e10f5f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-11T10:08:10.966 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:10.981+0000 7f94e10f5f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-11T10:08:11.018 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.033+0000 7f94e10f5f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-11T10:08:11.052 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-11T10:08:11.055 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:08:11.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-11T10:08:11.063 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.077+0000 7f94e10f5f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-11T10:08:11.084 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-01-11T10:08:11.084 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-11T10:08:11.116 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.131+0000 7f94e10f5f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-11T10:08:11.152 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:08:11.152 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-11T10:08:11.158 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.173+0000 7f94e10f5f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-11T10:08:11.179 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-01-11T10:08:11.179 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-11T10:08:11.246 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 84946e10-ef33-422d-b0b6-4df045330a9e 0 2024-01-11T10:08:11.265 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.279+0000 7f94e10f5f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-11T10:08:11.339 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.354+0000 7f94e10f5f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-11T10:08:11.370 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:11.379+0000 7f889726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:11.373 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:11.388+0000 7f889726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:11.388 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.402+0000 7f94e10f5f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-11T10:08:11.441 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.456+0000 7f94e10f5f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-11T10:08:11.452 INFO:teuthology.orchestra.run.smithi134.stdout:0 2024-01-11T10:08:11.464 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 768c1909-a443-48b3-b717-09915a9cf699 1 2024-01-11T10:08:11.492 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.506+0000 7f94e10f5f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-11T10:08:11.570 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:11.579+0000 7f0f57813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:11.573 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:11.588+0000 7f0f57813640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:11.651 INFO:teuthology.orchestra.run.smithi134.stdout:1 2024-01-11T10:08:11.662 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 0c527b7e-b40c-4aa3-b687-86994a574c12 2 2024-01-11T10:08:11.809 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:11.818+0000 7f4862cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:11.811 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:11.826+0000 7f4862cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:11.890 INFO:teuthology.orchestra.run.smithi134.stdout:2 2024-01-11T10:08:11.890 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:11.905+0000 7f94e10f5f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-11T10:08:11.901 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd new 81aa098c-cd18-4ccf-8f28-2cb59b10f1b6 3 2024-01-11T10:08:12.006 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:12.015+0000 7f72df31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:12.009 INFO:teuthology.orchestra.run.smithi134.stderr:2024-01-11T10:08:12.023+0000 7f72df31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:12.023 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.038+0000 7f94e10f5f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-11T10:08:12.089 INFO:teuthology.orchestra.run.smithi134.stdout:3 2024-01-11T10:08:12.100 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-01-11T10:08:12.137 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-11T10:08:12.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-01-11T10:08:12.143 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.157+0000 7f94e10f5f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-11T10:08:12.180 INFO:tasks.ceph.osd.0:Started 2024-01-11T10:08:12.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-01-11T10:08:12.188 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.202+0000 7f94e10f5f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-11T10:08:12.216 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-11T10:08:12.216 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-01-11T10:08:12.259 INFO:tasks.ceph.osd.1:Started 2024-01-11T10:08:12.259 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-01-11T10:08:12.285 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.299+0000 7f94e10f5f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-11T10:08:12.292 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-11T10:08:12.292 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-01-11T10:08:12.334 INFO:tasks.ceph.osd.2:Started 2024-01-11T10:08:12.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2024-01-11T10:08:12.365 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-11T10:08:12.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2024-01-11T10:08:12.384 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.399+0000 7f94e10f5f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-11T10:08:12.386 INFO:tasks.ceph.osd.3:Started 2024-01-11T10:08:12.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T10:08:12.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:12.520+0000 7f14e895e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:12.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:12.529+0000 7f14e895e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:12.547 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.561+0000 7f94e10f5f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-11T10:08:12.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:12.585 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":7,"fsid":"7d21cdfd-ee8f-43a7-9a95-045b41fc3d40","created":"2024-01-11T10:08:09.968587+0000","modified":"2024-01-11T10:08:12.102105+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T10:08:12.102105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"84946e10-ef33-422d-b0b6-4df045330a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"768c1909-a443-48b3-b717-09915a9cf699","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c527b7e-b40c-4aa3-b687-86994a574c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81aa098c-cd18-4ccf-8f28-2cb59b10f1b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T10:08:12.598 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.613+0000 7f94e10f5f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-11T10:08:12.599 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-11T10:08:12.599 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-11T10:08:12.763 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:12.778+0000 7f94e10f5f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-11T10:08:13.003 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T10:08:13.101 INFO:teuthology.misc.health.smithi040.stderr:2024-01-11T10:08:13.112+0000 7f17bad15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:13.105 INFO:teuthology.misc.health.smithi040.stderr:2024-01-11T10:08:13.119+0000 7f17bad15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:13.175 INFO:teuthology.misc.health.smithi040.stdout: 2024-01-11T10:08:13.175 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":7,"fsid":"7d21cdfd-ee8f-43a7-9a95-045b41fc3d40","created":"2024-01-11T10:08:09.968587+0000","modified":"2024-01-11T10:08:12.102105+0000","last_up_change":"0.000000","last_in_change":"2024-01-11T10:08:12.102105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"84946e10-ef33-422d-b0b6-4df045330a9e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"768c1909-a443-48b3-b717-09915a9cf699","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0c527b7e-b40c-4aa3-b687-86994a574c12","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"81aa098c-cd18-4ccf-8f28-2cb59b10f1b6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T10:08:13.185 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-01-11T10:08:13.629 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:13.643+0000 7f94e10f5f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-11T10:08:13.911 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:13.923+0000 7f94e10f5f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-11T10:08:13.963 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:13.977+0000 7f94e10f5f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-11T10:08:14.067 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:14.082+0000 7f94e10f5f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-11T10:08:15.088 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:08:15.103+0000 7f94816e1640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-11T10:08:19.589 DEBUG:teuthology.orchestra.run.smithi040:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-11T10:08:19.697 INFO:teuthology.misc.health.smithi040.stderr:2024-01-11T10:08:19.708+0000 7f66394ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:19.700 INFO:teuthology.misc.health.smithi040.stderr:2024-01-11T10:08:19.715+0000 7f66394ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:19.888 INFO:teuthology.misc.health.smithi040.stdout: 2024-01-11T10:08:19.888 INFO:teuthology.misc.health.smithi040.stdout:{"epoch":10,"fsid":"7d21cdfd-ee8f-43a7-9a95-045b41fc3d40","created":"2024-01-11T10:08:09.968587+0000","modified":"2024-01-11T10:08:19.109457+0000","last_up_change":"2024-01-11T10:08:17.104293+0000","last_in_change":"2024-01-11T10:08:12.102105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T10:08:18.140314+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84946e10-ef33-422d-b0b6-4df045330a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":1853162855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":1853162855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1853162855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":1853162855}]},"public_addr":"172.21.15.40:6801/1853162855","cluster_addr":"172.21.15.40:6800/1853162855","heartbeat_back_addr":"172.21.15.40:6804/1853162855","heartbeat_front_addr":"172.21.15.40:6805/1853162855","state":["exists","up"]},{"osd":1,"uuid":"768c1909-a443-48b3-b717-09915a9cf699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3832128222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":3832128222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3832128222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3832128222}]},"public_addr":"172.21.15.40:6802/3832128222","cluster_addr":"172.21.15.40:6803/3832128222","heartbeat_back_addr":"172.21.15.40:6806/3832128222","heartbeat_front_addr":"172.21.15.40:6807/3832128222","state":["exists","up"]},{"osd":2,"uuid":"0c527b7e-b40c-4aa3-b687-86994a574c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3707224435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6803","nonce":3707224435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3707224435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6807","nonce":3707224435}]},"public_addr":"172.21.15.134:6802/3707224435","cluster_addr":"172.21.15.134:6803/3707224435","heartbeat_back_addr":"172.21.15.134:6806/3707224435","heartbeat_front_addr":"172.21.15.134:6807/3707224435","state":["exists","up"]},{"osd":3,"uuid":"81aa098c-cd18-4ccf-8f28-2cb59b10f1b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6805","nonce":3625386763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3625386763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3625386763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6809","nonce":3625386763}]},"public_addr":"172.21.15.134:6805/3625386763","cluster_addr":"172.21.15.134:6804/3625386763","heartbeat_back_addr":"172.21.15.134:6808/3625386763","heartbeat_front_addr":"172.21.15.134:6809/3625386763","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T10:08:19.899 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-01-11T10:08:19.899 INFO:tasks.ceph:Creating RBD pool 2024-01-11T10:08:19.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-11T10:08:20.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:20.013+0000 7f61fd2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:20.009 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:20.023+0000 7f61fd2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:21.101 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'rbd' created 2024-01-11T10:08:21.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-11T10:08:21.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:21.232+0000 7f5fa339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:21.229 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:21.244+0000 7f5fa339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:22.106 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-11T10:08:22.122 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-11T10:08:22.123 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-11T10:08:22.123 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-11T10:08:22.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-11T10:08:22.124 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-11T10:08:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:22.267+0000 7f9a4d11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:22.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:22.274+0000 7f9a4d11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:22.496 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:22.507 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":4,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6800","nonce":3680903476},{"type":"v1","addr":"172.21.15.134:6801","nonce":3680903476}]},"active_addr":"172.21.15.134:6801/3680903476","active_change":"2024-01-11T10:08:14.095879+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":1329895979}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":853432608}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.134:0","nonce":2937997039}]}]} 2024-01-11T10:08:22.511 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-11T10:08:22.511 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-11T10:08:22.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T10:08:22.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:22.628+0000 7f122191e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:22.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:22.634+0000 7f122191e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:22.807 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:22.807 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":13,"fsid":"7d21cdfd-ee8f-43a7-9a95-045b41fc3d40","created":"2024-01-11T10:08:09.968587+0000","modified":"2024-01-11T10:08:22.119558+0000","last_up_change":"2024-01-11T10:08:17.104293+0000","last_in_change":"2024-01-11T10:08:12.102105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T10:08:18.140314+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-11T10:08:20.230191+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84946e10-ef33-422d-b0b6-4df045330a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":1853162855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":1853162855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1853162855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":1853162855}]},"public_addr":"172.21.15.40:6801/1853162855","cluster_addr":"172.21.15.40:6800/1853162855","heartbeat_back_addr":"172.21.15.40:6804/1853162855","heartbeat_front_addr":"172.21.15.40:6805/1853162855","state":["exists","up"]},{"osd":1,"uuid":"768c1909-a443-48b3-b717-09915a9cf699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3832128222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":3832128222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3832128222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3832128222}]},"public_addr":"172.21.15.40:6802/3832128222","cluster_addr":"172.21.15.40:6803/3832128222","heartbeat_back_addr":"172.21.15.40:6806/3832128222","heartbeat_front_addr":"172.21.15.40:6807/3832128222","state":["exists","up"]},{"osd":2,"uuid":"0c527b7e-b40c-4aa3-b687-86994a574c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3707224435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6803","nonce":3707224435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3707224435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6807","nonce":3707224435}]},"public_addr":"172.21.15.134:6802/3707224435","cluster_addr":"172.21.15.134:6803/3707224435","heartbeat_back_addr":"172.21.15.134:6806/3707224435","heartbeat_front_addr":"172.21.15.134:6807/3707224435","state":["exists","up"]},{"osd":3,"uuid":"81aa098c-cd18-4ccf-8f28-2cb59b10f1b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6805","nonce":3625386763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3625386763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3625386763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6809","nonce":3625386763}]},"public_addr":"172.21.15.134:6805/3625386763","cluster_addr":"172.21.15.134:6804/3625386763","heartbeat_back_addr":"172.21.15.134:6808/3625386763","heartbeat_front_addr":"172.21.15.134:6809/3625386763","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T10:08:22.819 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-11T10:08:22.820 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T10:08:22.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:22.938+0000 7f299c15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:22.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:22.944+0000 7f299c15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.117 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:23.117 INFO:teuthology.orchestra.run.smithi040.stdout:{"epoch":14,"fsid":"7d21cdfd-ee8f-43a7-9a95-045b41fc3d40","created":"2024-01-11T10:08:09.968587+0000","modified":"2024-01-11T10:08:23.122429+0000","last_up_change":"2024-01-11T10:08:17.104293+0000","last_in_change":"2024-01-11T10:08:12.102105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-11T10:08:18.140314+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-11T10:08:20.230191+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"84946e10-ef33-422d-b0b6-4df045330a9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6801","nonce":1853162855}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6800","nonce":1853162855}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6804","nonce":1853162855}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6805","nonce":1853162855}]},"public_addr":"172.21.15.40:6801/1853162855","cluster_addr":"172.21.15.40:6800/1853162855","heartbeat_back_addr":"172.21.15.40:6804/1853162855","heartbeat_front_addr":"172.21.15.40:6805/1853162855","state":["exists","up"]},{"osd":1,"uuid":"768c1909-a443-48b3-b717-09915a9cf699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6802","nonce":3832128222}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6803","nonce":3832128222}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6806","nonce":3832128222}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.40:6807","nonce":3832128222}]},"public_addr":"172.21.15.40:6802/3832128222","cluster_addr":"172.21.15.40:6803/3832128222","heartbeat_back_addr":"172.21.15.40:6806/3832128222","heartbeat_front_addr":"172.21.15.40:6807/3832128222","state":["exists","up"]},{"osd":2,"uuid":"0c527b7e-b40c-4aa3-b687-86994a574c12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6802","nonce":3707224435}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6803","nonce":3707224435}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6806","nonce":3707224435}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6807","nonce":3707224435}]},"public_addr":"172.21.15.134:6802/3707224435","cluster_addr":"172.21.15.134:6803/3707224435","heartbeat_back_addr":"172.21.15.134:6806/3707224435","heartbeat_front_addr":"172.21.15.134:6807/3707224435","state":["exists","up"]},{"osd":3,"uuid":"81aa098c-cd18-4ccf-8f28-2cb59b10f1b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6805","nonce":3625386763}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6804","nonce":3625386763}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6808","nonce":3625386763}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.134:6809","nonce":3625386763}]},"public_addr":"172.21.15.134:6805/3625386763","cluster_addr":"172.21.15.134:6804/3625386763","heartbeat_back_addr":"172.21.15.134:6808/3625386763","heartbeat_front_addr":"172.21.15.134:6809/3625386763","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-11T10:08:23.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-11T10:08:23.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-11T10:08:23.130 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-11T10:08:23.131 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-11T10:08:23.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.259+0000 7fae82f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.259+0000 7f68cc19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.259+0000 7fede34eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.250 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.259+0000 7fa0f3ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.267+0000 7fa0f3ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.267+0000 7fae82f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.267+0000 7f68cc19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.254 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.268+0000 7fede34eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.285 INFO:teuthology.orchestra.run.smithi040.stdout:38654705670 2024-01-11T10:08:23.285 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T10:08:23.286 INFO:teuthology.orchestra.run.smithi040.stdout:38654705670 2024-01-11T10:08:23.286 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-11T10:08:23.289 INFO:teuthology.orchestra.run.smithi040.stdout:38654705670 2024-01-11T10:08:23.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T10:08:23.291 INFO:teuthology.orchestra.run.smithi040.stdout:38654705670 2024-01-11T10:08:23.292 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T10:08:23.398 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.408+0000 7f565ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.399 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.408+0000 7f4497d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.416+0000 7f4497d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.417+0000 7f565ec6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.451 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.460+0000 7fda6d99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.454 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.468+0000 7fda6d99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.464 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.473+0000 7fe50202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.469 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:23.483+0000 7fe50202d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:23.602 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:23.620 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-01-11T10:08:23.626 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:23.638 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-01-11T10:08:23.654 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:23.665 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:23.667 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-01-11T10:08:23.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-01-11T10:08:24.621 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-11T10:08:24.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T10:08:24.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T10:08:24.678 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T10:08:24.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.756+0000 7fb7f0708640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.751 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.766+0000 7fb7f0708640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.785 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.794+0000 7f50c926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.804+0000 7f50c926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.834+0000 7f16f939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.842+0000 7f16f939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.838+0000 7f8904b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:24.847+0000 7f8904b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:24.950 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:24.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-01-11T10:08:24.989 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:25.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-01-11T10:08:25.021 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:25.026 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:25.033 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-01-11T10:08:25.038 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-01-11T10:08:25.968 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-11T10:08:26.003 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T10:08:26.034 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T10:08:26.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T10:08:26.092 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.098+0000 7fe914cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.096 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.110+0000 7fe914cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.153+0000 7fd37f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.162+0000 7fd37f06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.183+0000 7f073be2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.191+0000 7f073be2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.196+0000 7f5f7355e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:26.206+0000 7f5f7355e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:26.298 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:26.314 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-01-11T10:08:26.340 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:26.353 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-01-11T10:08:26.376 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:26.379 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:26.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-01-11T10:08:26.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-01-11T10:08:27.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-11T10:08:27.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T10:08:27.390 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T10:08:27.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T10:08:27.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.442+0000 7f5af44f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.452+0000 7f5af44f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.483+0000 7f04aa79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.492+0000 7f04aa79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.526+0000 7fc5ce26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.533+0000 7fc5ce26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.561+0000 7f3ede75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.554 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:27.569+0000 7f3ede75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:27.679 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:27.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.0 2024-01-11T10:08:27.702 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:27.715 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.3 2024-01-11T10:08:27.716 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:27.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.2 2024-01-11T10:08:27.775 INFO:teuthology.orchestra.run.smithi040.stdout:0 2024-01-11T10:08:27.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 0 for osd.1 2024-01-11T10:08:28.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-11T10:08:28.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-11T10:08:28.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-11T10:08:28.789 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-11T10:08:28.807 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.817+0000 7f7cbb304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.811 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.826+0000 7f7cbb304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.861+0000 7fd1c9ef7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.863+0000 7f533a79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.870+0000 7fd1c9ef7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.872+0000 7f533a79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.914+0000 7f3cece24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:28.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:28.922+0000 7f3cece24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:29.016 INFO:teuthology.orchestra.run.smithi040.stdout:38654705675 2024-01-11T10:08:29.034 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705675 for osd.0 2024-01-11T10:08:29.035 DEBUG:teuthology.parallel:result is None 2024-01-11T10:08:29.055 INFO:teuthology.orchestra.run.smithi040.stdout:38654705673 2024-01-11T10:08:29.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705673 for osd.3 2024-01-11T10:08:29.068 DEBUG:teuthology.parallel:result is None 2024-01-11T10:08:29.069 INFO:teuthology.orchestra.run.smithi040.stdout:38654705675 2024-01-11T10:08:29.082 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705675 for osd.2 2024-01-11T10:08:29.083 DEBUG:teuthology.parallel:result is None 2024-01-11T10:08:29.100 INFO:teuthology.orchestra.run.smithi040.stdout:38654705674 2024-01-11T10:08:29.111 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705670 got 38654705674 for osd.1 2024-01-11T10:08:29.112 DEBUG:teuthology.parallel:result is None 2024-01-11T10:08:29.112 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-11T10:08:29.112 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:08:29.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:29.269+0000 7f152499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:29.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:29.275+0000 7f152499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:29.442 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:29.443 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:08:29.454 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-11T10:08:28.136657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107944,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379416,"statfs":{"total":386547056640,"available":386436521984,"internally_reserved":0,"allocated":651264,"data_stored":1889666,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.012940"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.219147+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:22.219147+0000","last_peered":"2024-01-11T10:08:22.219147+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:22.219147+0000","last_undegraded":"2024-01-11T10:08:22.219147+0000","last_fullsized":"2024-01-11T10:08:22.219147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121810+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:28.121810+0000","last_peered":"2024-01-11T10:08:28.121810+0000","last_clean":"2024-01-11T10:08:28.121810+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:28.121810+0000","last_undegraded":"2024-01-11T10:08:28.121810+0000","last_fullsized":"2024-01-11T10:08:28.121810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.140053+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:23.140053+0000","last_peered":"2024-01-11T10:08:23.140053+0000","last_clean":"2024-01-11T10:08:23.140053+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:23.140053+0000","last_undegraded":"2024-01-11T10:08:23.140053+0000","last_fullsized":"2024-01-11T10:08:23.140053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.123182+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:28.123182+0000","last_peered":"2024-01-11T10:08:28.123182+0000","last_clean":"2024-01-11T10:08:28.123182+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:28.123182+0000","last_undegraded":"2024-01-11T10:08:28.123182+0000","last_fullsized":"2024-01-11T10:08:28.123182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.120901+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:28.120901+0000","last_peered":"2024-01-11T10:08:28.120901+0000","last_clean":"2024-01-11T10:08:28.120901+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:28.120901+0000","last_undegraded":"2024-01-11T10:08:28.120901+0000","last_fullsized":"2024-01-11T10:08:28.120901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.200324+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:22.200324+0000","last_peered":"2024-01-11T10:08:22.200324+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:22.200324+0000","last_undegraded":"2024-01-11T10:08:22.200324+0000","last_fullsized":"2024-01-11T10:08:22.200324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121423+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:28.121423+0000","last_peered":"2024-01-11T10:08:28.121423+0000","last_clean":"2024-01-11T10:08:28.121423+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:28.121423+0000","last_undegraded":"2024-01-11T10:08:28.121423+0000","last_fullsized":"2024-01-11T10:08:28.121423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.228160+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:22.228160+0000","last_peered":"2024-01-11T10:08:22.228160+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:22.228160+0000","last_undegraded":"2024-01-11T10:08:22.228160+0000","last_fullsized":"2024-01-11T10:08:22.228160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.134231+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:23.134231+0000","last_peered":"2024-01-11T10:08:23.134231+0000","last_clean":"2024-01-11T10:08:23.134231+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:23.134231+0000","last_undegraded":"2024-01-11T10:08:23.134231+0000","last_fullsized":"2024-01-11T10:08:23.134231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:08:29.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:08:29.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:29.573+0000 7f62eab04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:29.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:29.579+0000 7f62eab04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:29.745 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:29.746 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:08:29.757 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-11T10:08:28.136657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107944,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379416,"statfs":{"total":386547056640,"available":386436521984,"internally_reserved":0,"allocated":651264,"data_stored":1889666,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.012940"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.219147+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:22.219147+0000","last_peered":"2024-01-11T10:08:22.219147+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:22.219147+0000","last_undegraded":"2024-01-11T10:08:22.219147+0000","last_fullsized":"2024-01-11T10:08:22.219147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121810+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:28.121810+0000","last_peered":"2024-01-11T10:08:28.121810+0000","last_clean":"2024-01-11T10:08:28.121810+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:28.121810+0000","last_undegraded":"2024-01-11T10:08:28.121810+0000","last_fullsized":"2024-01-11T10:08:28.121810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.140053+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:23.140053+0000","last_peered":"2024-01-11T10:08:23.140053+0000","last_clean":"2024-01-11T10:08:23.140053+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:23.140053+0000","last_undegraded":"2024-01-11T10:08:23.140053+0000","last_fullsized":"2024-01-11T10:08:23.140053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.123182+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:28.123182+0000","last_peered":"2024-01-11T10:08:28.123182+0000","last_clean":"2024-01-11T10:08:28.123182+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:28.123182+0000","last_undegraded":"2024-01-11T10:08:28.123182+0000","last_fullsized":"2024-01-11T10:08:28.123182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.120901+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:28.120901+0000","last_peered":"2024-01-11T10:08:28.120901+0000","last_clean":"2024-01-11T10:08:28.120901+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:28.120901+0000","last_undegraded":"2024-01-11T10:08:28.120901+0000","last_fullsized":"2024-01-11T10:08:28.120901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.200324+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:22.200324+0000","last_peered":"2024-01-11T10:08:22.200324+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:22.200324+0000","last_undegraded":"2024-01-11T10:08:22.200324+0000","last_fullsized":"2024-01-11T10:08:22.200324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121423+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:28.121423+0000","last_peered":"2024-01-11T10:08:28.121423+0000","last_clean":"2024-01-11T10:08:28.121423+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:28.121423+0000","last_undegraded":"2024-01-11T10:08:28.121423+0000","last_fullsized":"2024-01-11T10:08:28.121423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.228160+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:22.228160+0000","last_peered":"2024-01-11T10:08:22.228160+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:22.228160+0000","last_undegraded":"2024-01-11T10:08:22.228160+0000","last_fullsized":"2024-01-11T10:08:22.228160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.134231+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:23.134231+0000","last_peered":"2024-01-11T10:08:23.134231+0000","last_clean":"2024-01-11T10:08:23.134231+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:23.134231+0000","last_undegraded":"2024-01-11T10:08:23.134231+0000","last_fullsized":"2024-01-11T10:08:23.134231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:08:29.758 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-11T10:08:29.758 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-11T10:08:29.758 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-11T10:08:29.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-11T10:08:29.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:29.876+0000 7f0945017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:29.866 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:29.881+0000 7f0945017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:30.093 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:30.093 INFO:teuthology.orchestra.run.smithi040.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-11T10:08:30.106 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-11T10:08:30.106 INFO:teuthology.run_tasks:Running task thrashosds... 2024-01-11T10:08:30.117 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-01-11T10:08:30.117 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-matanb-crimson-multicore-msgr'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd failsafe full ratio': 0.95, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '4e3957a13c3b8938fe41053b13fb80161e1916b7'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'f14f60afd567d61490bfe92caba04c0ffb5b09d2'}} 2024-01-11T10:08:30.117 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_down': 0, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2024-01-11T10:08:30.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-01-11T10:08:30.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:30.241+0000 7f3f48c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:30.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:30.251+0000 7f3f48c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:30.434 INFO:tasks.thrashosds:Beginning thrashosds... 2024-01-11T10:08:30.435 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-11T10:08:30.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:08:30.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:30.553+0000 7fa72f35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:30.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:30.559+0000 7fa72f35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:30.728 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:30.729 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:08:30.740 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-01-11T10:08:30.100422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107944,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379416,"statfs":{"total":386547056640,"available":386436521984,"internally_reserved":0,"allocated":651264,"data_stored":1889666,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.976706"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.219147+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:22.219147+0000","last_peered":"2024-01-11T10:08:22.219147+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:22.219147+0000","last_undegraded":"2024-01-11T10:08:22.219147+0000","last_fullsized":"2024-01-11T10:08:22.219147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121810+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:28.121810+0000","last_peered":"2024-01-11T10:08:28.121810+0000","last_clean":"2024-01-11T10:08:28.121810+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:28.121810+0000","last_undegraded":"2024-01-11T10:08:28.121810+0000","last_fullsized":"2024-01-11T10:08:28.121810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.140053+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:23.140053+0000","last_peered":"2024-01-11T10:08:23.140053+0000","last_clean":"2024-01-11T10:08:23.140053+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:23.140053+0000","last_undegraded":"2024-01-11T10:08:23.140053+0000","last_fullsized":"2024-01-11T10:08:23.140053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.123182+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:28.123182+0000","last_peered":"2024-01-11T10:08:28.123182+0000","last_clean":"2024-01-11T10:08:28.123182+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:28.123182+0000","last_undegraded":"2024-01-11T10:08:28.123182+0000","last_fullsized":"2024-01-11T10:08:28.123182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.120901+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:28.120901+0000","last_peered":"2024-01-11T10:08:28.120901+0000","last_clean":"2024-01-11T10:08:28.120901+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:28.120901+0000","last_undegraded":"2024-01-11T10:08:28.120901+0000","last_fullsized":"2024-01-11T10:08:28.120901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.200324+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:22.200324+0000","last_peered":"2024-01-11T10:08:22.200324+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:22.200324+0000","last_undegraded":"2024-01-11T10:08:22.200324+0000","last_fullsized":"2024-01-11T10:08:22.200324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121423+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:28.121423+0000","last_peered":"2024-01-11T10:08:28.121423+0000","last_clean":"2024-01-11T10:08:28.121423+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:28.121423+0000","last_undegraded":"2024-01-11T10:08:28.121423+0000","last_fullsized":"2024-01-11T10:08:28.121423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.228160+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:22.228160+0000","last_peered":"2024-01-11T10:08:22.228160+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:22.228160+0000","last_undegraded":"2024-01-11T10:08:22.228160+0000","last_fullsized":"2024-01-11T10:08:22.228160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.134231+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:23.134231+0000","last_peered":"2024-01-11T10:08:23.134231+0000","last_clean":"2024-01-11T10:08:23.134231+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:23.134231+0000","last_undegraded":"2024-01-11T10:08:23.134231+0000","last_fullsized":"2024-01-11T10:08:23.134231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:08:30.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:08:30.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:30.859+0000 7f11a9b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:30.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:30.865+0000 7f11a9b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.032 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:31.033 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:08:31.044 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-01-11T10:08:30.100422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":107944,"kb_used_data":636,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377379416,"statfs":{"total":386547056640,"available":386436521984,"internally_reserved":0,"allocated":651264,"data_stored":1889666,"data_compressed":89901,"data_compressed_allocated":122880,"data_compressed_original":1720320,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.976706"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.219147+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:22.219147+0000","last_peered":"2024-01-11T10:08:22.219147+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:22.219147+0000","last_undegraded":"2024-01-11T10:08:22.219147+0000","last_fullsized":"2024-01-11T10:08:22.219147+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121810+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:28.121810+0000","last_peered":"2024-01-11T10:08:28.121810+0000","last_clean":"2024-01-11T10:08:28.121810+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:28.121810+0000","last_undegraded":"2024-01-11T10:08:28.121810+0000","last_fullsized":"2024-01-11T10:08:28.121810+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.140053+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:23.140053+0000","last_peered":"2024-01-11T10:08:23.140053+0000","last_clean":"2024-01-11T10:08:23.140053+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:23.140053+0000","last_undegraded":"2024-01-11T10:08:23.140053+0000","last_fullsized":"2024-01-11T10:08:23.140053+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.123182+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:28.123182+0000","last_peered":"2024-01-11T10:08:28.123182+0000","last_clean":"2024-01-11T10:08:28.123182+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:28.123182+0000","last_undegraded":"2024-01-11T10:08:28.123182+0000","last_fullsized":"2024-01-11T10:08:28.123182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.120901+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:28.120901+0000","last_peered":"2024-01-11T10:08:28.120901+0000","last_clean":"2024-01-11T10:08:28.120901+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:28.120901+0000","last_undegraded":"2024-01-11T10:08:28.120901+0000","last_fullsized":"2024-01-11T10:08:28.120901+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.200324+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:22.200324+0000","last_peered":"2024-01-11T10:08:22.200324+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:22.200324+0000","last_undegraded":"2024-01-11T10:08:22.200324+0000","last_fullsized":"2024-01-11T10:08:22.200324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:28.121423+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:28.121423+0000","last_peered":"2024-01-11T10:08:28.121423+0000","last_clean":"2024-01-11T10:08:28.121423+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:28.121423+0000","last_undegraded":"2024-01-11T10:08:28.121423+0000","last_fullsized":"2024-01-11T10:08:28.121423+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-01-11T10:08:22.228160+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:22.228160+0000","last_peered":"2024-01-11T10:08:22.228160+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:22.228160+0000","last_undegraded":"2024-01-11T10:08:22.228160+0000","last_fullsized":"2024-01-11T10:08:22.228160+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":142,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-01-11T10:08:23.134231+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:23.134231+0000","last_peered":"2024-01-11T10:08:23.134231+0000","last_clean":"2024-01-11T10:08:23.134231+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:23.134231+0000","last_undegraded":"2024-01-11T10:08:23.134231+0000","last_fullsized":"2024-01-11T10:08:23.134231+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705675,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27000,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344840,"statfs":{"total":96636764160,"available":96609116160,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27012,"kb_used_data":176,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344828,"statfs":{"total":96636764160,"available":96609103872,"internally_reserved":0,"allocated":180224,"data_stored":620811,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705674,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":27233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:08:31.045 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-11T10:08:31.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-01-11T10:08:31.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.164+0000 7fbabe26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.170+0000 7fbabe26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.340 INFO:teuthology.orchestra.run.smithi040.stdout:epoch 14 2024-01-11T10:08:31.340 INFO:teuthology.orchestra.run.smithi040.stdout:fsid 7d21cdfd-ee8f-43a7-9a95-045b41fc3d40 2024-01-11T10:08:31.340 INFO:teuthology.orchestra.run.smithi040.stdout:created 2024-01-11T10:08:09.968587+0000 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:modified 2024-01-11T10:08:23.122429+0000 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:crush_version 3 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:full_ratio 0.95 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:backfillfull_ratio 0.9 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:nearfull_ratio 0.85 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:require_min_compat_client luminous 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:min_compat_client jewel 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:require_osd_release squid 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:stretch_mode_enabled false 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:allow_crimson=true 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 12 flags hashpspool,nopgchange,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,nopgchange,crimson stripe_width 0 application rbd read_balance_score 1.50 2024-01-11T10:08:31.341 INFO:teuthology.orchestra.run.smithi040.stdout:max_osd 4 2024-01-11T10:08:31.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.40:6801/1853162855 v2:172.21.15.40:6800/1853162855 exists,up 84946e10-ef33-422d-b0b6-4df045330a9e 2024-01-11T10:08:31.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.40:6802/3832128222 v2:172.21.15.40:6803/3832128222 exists,up 768c1909-a443-48b3-b717-09915a9cf699 2024-01-11T10:08:31.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.134:6802/3707224435 v2:172.21.15.134:6803/3707224435 exists,up 0c527b7e-b40c-4aa3-b687-86994a574c12 2024-01-11T10:08:31.342 INFO:teuthology.orchestra.run.smithi040.stdout:osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.134:6805/3625386763 v2:172.21.15.134:6804/3625386763 exists,up 81aa098c-cd18-4ccf-8f28-2cb59b10f1b6 2024-01-11T10:08:31.352 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.40:6801/1853162855 v2:172.21.15.40:6800/1853162855 exists,up 84946e10-ef33-422d-b0b6-4df045330a9e', 'osd.1 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.40:6802/3832128222 v2:172.21.15.40:6803/3832128222 exists,up 768c1909-a443-48b3-b717-09915a9cf699', 'osd.2 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.134:6802/3707224435 v2:172.21.15.134:6803/3707224435 exists,up 0c527b7e-b40c-4aa3-b687-86994a574c12', 'osd.3 up in weight 1 up_from 9 up_thru 12 down_at 0 last_clean_interval [0,0) v2:172.21.15.134:6805/3625386763 v2:172.21.15.134:6804/3625386763 exists,up 81aa098c-cd18-4ccf-8f28-2cb59b10f1b6'] 2024-01-11T10:08:31.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-01-11T10:08:31.459 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-01-11T10:08:31.459 INFO:teuthology.orchestra.run.smithi040.stdout: "name": "mon.a", 2024-01-11T10:08:31.459 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster": "ceph", 2024-01-11T10:08:31.459 INFO:teuthology.orchestra.run.smithi040.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-01-11T10:08:31.459 INFO:teuthology.orchestra.run.smithi040.stdout: "admin_socket_mode": "", 2024-01-11T10:08:31.459 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_client_required": "cephx, none", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_cluster_required": "cephx", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_debug": "true", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_service_required": "cephx", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_service_ticket_ttl": "240.000000", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "auth_supported": "cephx", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio": "true", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio_max_queue_depth": "1024", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio_poll_ms": "250", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_aio_reap_max": "16", 2024-01-11T10:08:31.460 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_async_discard": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_block_size": "4096", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_aio": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_debug_inflight_ios": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_enable_discard": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_flock_retry": "3", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_flock_retry_interval": "0.100000", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_inject_crash": "0", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_inject_crash_flush_delay": "2", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_ioring": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_ioring_hipri": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_ioring_sqthread_poll": "false", 2024-01-11T10:08:31.461 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_read_buffer_alignment": "4096", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bdev_type": "", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_alloc_size": "1048576", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_allocator": "hybrid", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_buffered_io": "true", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_check_for_zeros": "false", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_check_volume_selector_often": "false", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_compact_log_sync": "false", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_log_compact_min_size": "16777216", 2024-01-11T10:08:31.462 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_log_replay_check_allocations": "true", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_max_log_runway": "4194304", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_max_prefetch": "1048576", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_min_flush_size": "524288", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_min_log_runway": "1048576", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_replay_recovery": "false", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_shared_alloc_size": "65536", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluefs_sync_write": "false", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_allocation_from_file": "true", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_allocator": "hybrid", 2024-01-11T10:08:31.463 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_blobid_prealloc": "10240", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_create": "true", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_db_create": "false", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_db_path": "", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_db_size": "0", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_path": "", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_preallocate_file": "false", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_size": "107374182400", 2024-01-11T10:08:31.464 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_wal_create": "false", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_wal_path": "", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_block_wal_size": "100663296", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs": "true", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs_env_mirror": "false", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_autotune": "true", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-01-11T10:08:31.465 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_size": "0", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cache_type": "2q", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_clone_cow": "true", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_algorithm": "snappy", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_max_blob_size": "0", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-01-11T10:08:31.466 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_min_blob_size": "0", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_mode": "none", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_csum_type": "crc32c", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_enforce_settings": "default", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_freelist": "false", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_fsck_abort": "false", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_inject_read_err": "false", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_legacy_omap": "false", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-01-11T10:08:31.467 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_prefill": "0.000000", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_small_allocations": "0", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_default_buffered_read": "true", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_default_buffered_write": "false", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_deferred_batch_ops": "0", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-01-11T10:08:31.468 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_elastic_shared_blobs": "true", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fail_eio": "false", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mkfs": "true", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mount": "false", 2024-01-11T10:08:31.469 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_umount": "false", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_ignore_data_csum": "false", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_kvbackend": "rocksdb", 2024-01-11T10:08:31.470 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_log_op_age": "5.000000", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_alloc_size": "0", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_blob_size": "0", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_defer_interval": "3.000000", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_max_deferred_txc": "32", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_min_alloc_size": "0", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_nid_prealloc": "1024", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_prefer_deferred_size": "0", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-01-11T10:08:31.471 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_qfsck_on_mount": "true", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_retry_disk_reads": "3", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_cf": "true", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_rocksdb_options_annex": "", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_coremask": "0x1", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_io_sleep": "5", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_max_io_completion": "0", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_spdk_mem": "512", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_sync_submit_transaction": "false", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_bytes": "67108864", 2024-01-11T10:08:31.472 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_cost_per_io": "0", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_tracing": "false", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_volume_selection_reserved": "0", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-01-11T10:08:31.473 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "bluestore_zero_block_detection": "false", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_action_update_interval": "2", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_mount_timeout": "10", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-01-11T10:08:31.474 INFO:teuthology.orchestra.run.smithi040.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_cluster_require_signatures": "false", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_cluster_require_version": "2", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_require_signatures": "false", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_require_version": "2", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_service_require_signatures": "false", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_service_require_version": "2", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "cephx_sign_messages": "true", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "chdir": "", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_acl_type": "", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_asio_thread_count": "2", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_cache_mid": "0.750000", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_cache_size": "16384", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_caps_release_delay": "5", 2024-01-11T10:08:31.475 INFO:teuthology.orchestra.run.smithi040.stdout: "client_check_pool_perm": "true", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_collect_and_send_global_metrics": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_debug_force_sync_read": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_debug_getattr_caps": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_debug_inject_tick_delay": "0", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_die_on_failed_remount": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_dirsize_rbytes": "true", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_force_lazyio": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_fs": "", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_inject_fixed_oldest_tid": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_inject_release_failure": "false", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_max_inline_size": "4096", 2024-01-11T10:08:31.476 INFO:teuthology.orchestra.run.smithi040.stdout: "client_max_retries_on_remount_failure": "5", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mds_namespace": "", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_metadata": "", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mount_gid": "-1", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mount_timeout": "300", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mount_uid": "-1", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_mountpoint": "/", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_notify_timeout": "10", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc": "true", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_max_dirty": "104857600", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_max_dirty_age": "5.000000", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_max_objects": "1000", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_size": "209715200", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_oc_target_dirty": "8388608", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_permissions": "true", 2024-01-11T10:08:31.477 INFO:teuthology.orchestra.run.smithi040.stdout: "client_quota": "true", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_quota_df": "true", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_readahead_max_bytes": "0", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_readahead_max_periods": "4", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_readahead_min": "131072", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_reconnect_stale": "false", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_shutdown_timeout": "30", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_snapdir": ".snap", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_tick_interval": "1", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_trace": "", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_try_dentry_invalidate": "false", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_use_faked_inos": "false", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "client_use_random_mds": "false", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_graylog": "false", 2024-01-11T10:08:31.478 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_graylog_port": "12201", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_monitors": "default=true", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_syslog": "false", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "clog_to_syslog_level": "info", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_addr": "-", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_network": "", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "cluster_network_interface": "", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zlib_isal": "false", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zlib_level": "5", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zlib_winsize": "-15", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "compressor_zstd_level": "1", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-01-11T10:08:31.479 INFO:teuthology.orchestra.run.smithi040.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_alien_op_num_threads": "6", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_alien_thread_cpu_cores": "", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_osd_obc_lru_size": "10", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crimson_seastar_smp": "1", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location": "", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location_hook": "", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "crush_location_hook_timeout": "10", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "daemonize": "false", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "daos_pool": "tank", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "dbstore_db_name_prefix": "dbstore", 2024-01-11T10:08:31.480 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_alienstore": "0/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_allow_any_pool_priority": "false", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_asok": "1/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_asok_assert_abort": "false", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_asserts_on_shutdown": "true", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_auth": "20/20", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_bdev": "1/3", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_bluefs": "1/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_bluestore": "1/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_buffer": "0/1", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_ceph_exporter": "1/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_cephfs_mirror": "0/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_cephsqlite": "0/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_client": "0/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_compressor": "1/5", 2024-01-11T10:08:31.481 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_context": "0/1", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_crimson_interrupt": "0/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_crush": "1/1", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_crypto": "1/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_cyanstore": "0/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_deliberately_leak_memory": "false", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_disable_randomized_ping": "false", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_dpdk": "1/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_eventtrace": "1/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_filer": "0/1", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_filestore": "1/3", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_finisher": "1/1", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_fuse": "1/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_heartbeat_testing_span": "0", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_heartbeatmap": "1/5", 2024-01-11T10:08:31.482 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_immutable_obj_cache": "0/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_javaclient": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_journal": "1/3", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_journaler": "0/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_kstore": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_lockdep": "0/1", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mclock": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_balancer": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_locker": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_log": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_log_expire": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mds_migrator": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_memstore": "1/5", 2024-01-11T10:08:31.483 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mgr": "2/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mgrc": "1/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_mon": "20/20", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_monc": "0/10", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_ms": "1/1", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_none": "0/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_objclass": "0/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_objectcacher": "0/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_objecter": "0/1", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_optracker": "0/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_osd": "1/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_paxos": "20/20", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_perfcounter": "1/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_prioritycache": "1/5", 2024-01-11T10:08:31.484 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rados": "0/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd": "0/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd_mirror": "0/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd_pwl": "0/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rbd_replay": "0/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_refs": "0/0", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_reserver": "1/1", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw": "1/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_access": "1/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_datacache": "1/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_dbstore": "1/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_flight": "1/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rgw_sync": "1/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_rocksdb": "4/5", 2024-01-11T10:08:31.485 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_backref": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_cache": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_cleaner": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_device": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_epm": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_journal": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_lba": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_odata": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_omap": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_onode": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_t": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_seastore_tm": "0/5", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_striper": "0/1", 2024-01-11T10:08:31.486 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_test": "0/5", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_throttle": "1/1", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_timer": "0/1", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_tp": "0/5", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "debug_trace": "1/5", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "device_failure_prediction_mode": "none", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_graylog": "false", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_journald": "false", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_stderr": "true", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "err_to_syslog": "false", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event_tracing": "false", 2024-01-11T10:08:31.487 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_addr": "0.0.0.0", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_http_port": "9926", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_prio_limit": "5", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_sort_metrics": "true", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "exporter_stats_period": "5", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "fake_statfs_for_testing": "0", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "fatal_signal_handlers": "true", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "filer_max_purge_ops": "10", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "filer_max_truncate_ops": "128", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_apply_finisher_threads": "1", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_blackhole": "false", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_btrfs_clone_range": "true", 2024-01-11T10:08:31.488 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_btrfs_snap": "true", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_caller_concurrency": "10", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_collect_device_partition_information": "true", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_commit_timeout": "600.000000", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_inject_read_err": "false", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_omap_check": "false", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_random_read_err": "0.000000", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_debug_verify_split": "false", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_dump_file": "", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fadvise": "true", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fail_eio": "true", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fd_cache_shards": "16", 2024-01-11T10:08:31.489 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fd_cache_size": "128", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fiemap": "false", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fiemap_threshold": "4096", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_index_retry_probability": "0.000000", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_inject_stall": "0", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_journal_parallel": "false", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_journal_trailing": "false", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_journal_writeahead": "false", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_kill_at": "0", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size": "0", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-01-11T10:08:31.490 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs": "0", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs_other": "2", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_sync_interval": "5.000000", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size": "0", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_merge_threshold": "-10", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_min_sync_interval": "0.010000", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_odsync_write": "false", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_omap_backend": "rocksdb", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_omap_backend_path": "", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_omap_header_cache_size": "1024", 2024-01-11T10:08:31.491 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_ondisk_finisher_threads": "1", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_op_thread_timeout": "60", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_op_threads": "2", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_punch_hole": "false", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-01-11T10:08:31.492 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_bytes": "104857600", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_queue_max_ops": "50", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_seek_data_hole": "false", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_sloppy_crc": "false", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_splice": "false", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_split_multiple": "2", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_split_rand_factor": "20", 2024-01-11T10:08:31.493 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_update_to": "1000", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_enable": "true", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-01-11T10:08:31.494 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_xfs_extsize": "false", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "filestore_zfs_snap": "false", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fio_dir": "/tmp/fio", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fsid": "7d21cdfd-ee8f-43a7-9a95-045b41fc3d40", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_allow_other": "true", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_atomic_o_trunc": "true", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_big_writes": "true", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_debug": "false", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_default_permissions": "false", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_disable_pagecache": "false", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_max_write": "0", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_multithreaded": "true", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_require_active_mds": "true", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_set_user_groups": "true", 2024-01-11T10:08:31.495 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_splice_move": "true", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_splice_read": "true", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_splice_write": "true", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_syncfs_on_mksnap": "true", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "fuse_use_invalidate_cb": "true", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "gss_target_name": "ceph", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_file": "", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_inject_failure": "0", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "heartbeat_interval": "5", 2024-01-11T10:08:31.496 INFO:teuthology.orchestra.run.smithi040.stdout: "host": "smithi040", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_max_size": "1073741824", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-01-11T10:08:31.497 INFO:teuthology.orchestra.run.smithi040.stdout: "immutable_object_cache_watermark": "0.900000", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "inject_early_sigterm": "false", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "jaeger_agent_port": "6799", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "jaeger_tracing_enable": "false", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_aio": "true", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_align_min_size": "65536", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_block_align": "true", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_block_size": "4096", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_dio": "true", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_discard": "false", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_force_aio": "false", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_ignore_corruption": "false", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_max_write_bytes": "10485760", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_max_write_entries": "100", 2024-01-11T10:08:31.498 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_replay_from": "0", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_high_multiple": "0.000000", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_throttle_max_multiple": "0.000000", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_write_header_frequency": "0", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journal_zero_on_create": "false", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journaler_prefetch_periods": "10", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journaler_prezero_periods": "5", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "journaler_write_head_interval": "15", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "key": "", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "keyfile": "", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_backend": "rocksdb", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_default_stripe_size": "65536", 2024-01-11T10:08:31.499 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_fsck_on_mount": "false", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_fsck_on_mount_deep": "true", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_max_bytes": "67108864", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_max_ops": "512", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_nid_prealloc": "1024", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_onode_map_size": "1024", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_sync_submit_transaction": "false", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "kstore_sync_transaction": "false", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "librados_thread_count": "2", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "lockdep": "false", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "lockdep_force_backtrace": "false", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "log_coarse_timestamps": "true", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-01-11T10:08:31.500 INFO:teuthology.orchestra.run.smithi040.stdout: "log_flush_on_exit": "false", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_graylog_host": "127.0.0.1", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_graylog_port": "12201", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_max_new": "1000", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_max_recent": "10000", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_stderr_prefix": "", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_stop_at_utilization": "0.970000", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_file": "true", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_graylog": "false", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_journald": "false", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_stderr": "false", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "log_to_syslog": "false", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "mandatory_topic_permissions": "false", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "max_rotating_auth_attempts": "10", 2024-01-11T10:08:31.501 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_action_on_write_error": "1", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_alternate_name_max": "8192", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_asio_thread_count": "2", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_export_pin": "true", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_dirs": "true", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_interval": "5", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_fragment_size_max": "100000", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_idle_threshold": "0.000000", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_interval": "10", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_max": "-1", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_max_until": "-1", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_merge_size": "50", 2024-01-11T10:08:31.502 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_midchunk": "0.300000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_min_rebalance": "0.100000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_min_start": "0.200000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_minchunk": "0.001000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_mode": "0", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_need_max": "1.200000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_need_min": "0.800000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_overload_epochs": "2", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_sample_interval": "3.000000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_bits": "3", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_rd": "25000.000000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_size": "10000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_split_wr": "10000.000000", 2024-01-11T10:08:31.503 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_target_decay": "10.000000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_beacon_grace": "15.000000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_beacon_interval": "4.000000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_beacon_mon_down_grace": "60", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_memory_limit": "4294967296", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_mid": "0.700000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_release_free_interval": "10", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_reservation": "0.050000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_trim_interval": "1", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cache_trim_threshold": "262144", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-01-11T10:08:31.504 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_client_delegate_inos_pct": "50", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_client_prealloc_inos": "1000", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_connect_bootstrapping": "false", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_damage_table_max_entries": "10000", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_auth_pins": "false", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_frag": "false", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_scatterstat": "false", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_debug_subtrees": "false", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_decay_halflife": "5.000000", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_default_dir_hash": "2", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_defer_session_stale": "true", 2024-01-11T10:08:31.505 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_deny_all_reconnect": "false", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_keys_per_op": "16384", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_max_commit_size": "10", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_max_entries": "0", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dir_prefetch": "true", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dirstat_min_interval": "1.000000", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_after_rejoin": "false", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_on_map": "false", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_threshold_file": "0", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_early_reply": "true", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_enable_op_tracker": "true", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_enforce_unique_name": "true", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_distributed": "true", 2024-01-11T10:08:31.506 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_random": "true", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_forward_all_requests_to_auth": "false", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_health_cache_threshold": "1.500000", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_health_summarize_threshold": "10", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_heartbeat_grace": "15.000000", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_heartbeat_reset_grace": "1000", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_health_dummy": "false", 2024-01-11T10:08:31.507 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_migrator_session_race": "false", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_join_fs": "", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_journal_format": "1", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_create_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_export_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_import_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_journal_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_journal_expire_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_journal_replay_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_link_at": "0", 2024-01-11T10:08:31.508 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_mdstable_at": "0", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_openc_at": "0", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_rename_at": "0", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_shutdown_at": "0", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_event_large_threshold": "524288", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_events_per_segment": "1024", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_major_segment_event_ratio": "12", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_max_events": "-1", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_max_segments": "128", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_pause": "false", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_segment_size": "0", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_skip_corrupt_events": "false", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_skip_unbounded_events": "false", 2024-01-11T10:08:31.509 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_log_warn_factor": "2.000000", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_caps_per_client": "1048576", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_completed_flushes": "100000", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_completed_requests": "100000", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_export_size": "20971520", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_file_recover": "32", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_purge_files": "64", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_purge_ops": "8192", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_max_snaps_per_dir": "100", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_metrics_update_interval": "2", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_min_caps_per_client": "100", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_min_caps_working_set": "10000", 2024-01-11T10:08:31.510 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_numa_node": "-1", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_complaint_time": "30.000000", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_duration": "600", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_size": "20", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_slow_op_size": "20", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_history_slow_op_threshold": "10", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_op_log_threshold": "5", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_ping_grace": "15", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_ping_interval": "5", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_max_caps": "30000", 2024-01-11T10:08:31.511 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_max_decay_threshold": "131072", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_recall_warning_threshold": "262144", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_reconnect_timeout": "45.000000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_replay_interval": "1.000000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_root_ino_gid": "0", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_root_ino_uid": "0", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_blocklist_on_evict": "true", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_blocklist_on_timeout": "true", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-01-11T10:08:31.512 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_session_metadata_threshold": "16777216", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_shutdown_check": "0", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_skip_ino": "0", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_sleep_rank_change": "0.000000", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_snap_max_uid": "4294967294", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_snap_min_uid": "0", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_snap_rstat": "false", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_standby_replay_damaged": "false", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_symlink_recovery": "true", 2024-01-11T10:08:31.513 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_task_status_update_interval": "2.000000", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_thrash_exports": "0", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_thrash_fragments": "0", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_tick_interval": "5.000000", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_valgrind_exit": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_verify_backtrace": "1", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_verify_scatter": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_wipe_ino_prealloc": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mds_wipe_sessions": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mempool_debug": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_debug_omit_block_device_write": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_device_bytes": "1073741824", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_page_set": "false", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "memstore_page_size": "65536", 2024-01-11T10:08:31.514 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_client_bytes": "134217728", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_client_messages": "512", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_connect_retry_interval": "1.000000", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_disabled_modules": "", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_max_pg_num_change": "128", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mds_bytes": "134217728", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mds_messages": "128", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mon_bytes": "134217728", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_mon_messages": "128", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_osd_bytes": "536870912", 2024-01-11T10:08:31.515 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_osd_messages": "8192", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_pool": "true", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_service_beacon_grace": "60.000000", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_standby_modules": "true", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_stats_period": "5", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_stats_threshold": "5", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_tick_period": "2", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_accept_timeout_factor": "2.000000", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_allow_pool_delete": "true", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_allow_pool_size_one": "true", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_auth_validate_all_caps": "true", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-01-11T10:08:31.516 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_bytes": "104857600", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_directed_command_retry": "2", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval": "3.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_hunt_parallel": "3", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_log_interval": "1.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_ping_interval": "10.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_ping_timeout": "30.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_client_target_rank": "-1", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_clock_drift_allowed": "1.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-01-11T10:08:31.517 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_file_level": "debug", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_file": "true", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_graylog": "false", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_journald": "false", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_stderr": "false", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_compact_on_bootstrap": "false", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_compact_on_start": "false", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_compact_on_trim": "true", 2024-01-11T10:08:31.518 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_con_tracker_persist_interval": "10", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_con_tracker_score_halflife": "43200", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_config_key_max_entry_size": "65536", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_cpu_threads": "4", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_crush_min_required_version": "hammer", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_daemon_bytes": "419430400", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data_avail_crit": "5", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data_avail_warn": "5", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_data_size_warn": "16106127360", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_block_osdmap_trim": "false", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_dump_json": "false", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-01-11T10:08:31.519 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_dump_transactions": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_extra_checks": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_require_reef": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_no_require_squid": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_delta_reset_interval": "10.000000", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_dns_srv_name": "ceph-mon", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_down_mkfs_grace": "120", 2024-01-11T10:08:31.520 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_election_default_strategy": "1", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_election_timeout": "5.000000", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_elector_ping_divisor": "2", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_elector_ping_timeout": "2.000000", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_enable_op_tracker": "true", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_fake_pool_delete": "false", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_force_quorum_join": "false", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_globalid_prealloc": "10000", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_detail_to_clog": "false", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_log_update_period": "5", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_max_detail": "50", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_to_clog": "true", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_to_clog_interval": "600", 2024-01-11T10:08:31.521 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_host": "172.21.15.40", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_host_override": "", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_initial_members": "", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_keyvaluedb": "rocksdb", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_lease": "5.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_log_full_interval": "50", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_log_max": "10000", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_log_max_summary": "50", 2024-01-11T10:08:31.522 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_log_entries_per_event": "4096", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_log_epochs": "500", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_mdsmap_epochs": "500", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_mgrmap_epochs": "500", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_osd": "10000", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_pg_per_osd": "10000", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_pool_pg_num": "65536", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mds_force_trim_to": "0", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mds_skip_sanity": "false", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_memory_autotune": "true", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_memory_target": "2147483648", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_beacon_grace": "30", 2024-01-11T10:08:31.523 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_digest_period": "5", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_inactive_grace": "60", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_mkfs_grace": "240", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_min_osdmap_epochs": "50", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_complaint_time": "30", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_duration": "600", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_size": "20", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_slow_op_size": "20", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_history_slow_op_threshold": "10", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_op_log_threshold": "5", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-01-11T10:08:31.524 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_auto_mark_in": "false", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_auto_mark_new_in": "true", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_cache_size": "500", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_cache_size_min": "134217728", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_crush_smoke_test": "true", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_destroyed_out_interval": "600", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_down_out_interval": "600", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_force_trim_to": "0", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_full_ratio": "0.950000", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-01-11T10:08:31.525 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_laggy_halflife": "3600", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_laggy_max_interval": "300", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_laggy_weight": "0.300000", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_max_creating_pgs": "1024", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_max_initial_pgs": "1024", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_min_down_reporters": "2", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_prime_pg_temp": "true", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_report_timeout": "900", 2024-01-11T10:08:31.526 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_warn_num_repaired": "10", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osd_warn_op_age": "32.000000", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_interval": "2", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_min": "15", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_stuck_threshold": "60", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_min_objects": "10000", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_min_per_osd": "0", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pool_quota_crit_threshold": "0", 2024-01-11T10:08:31.527 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_pool_quota_warn_threshold": "0", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_probe_timeout": "2.000000", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_max_change": "0.050000", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_max_osds": "4", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_interval": "86400", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_max_keys": "100", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_scrub_timeout": "300", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_session_timeout": "300", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_smart_report_timeout": "5", 2024-01-11T10:08:31.528 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stat_smooth_intervals": "6", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_pool_min_size": "2", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_pool_size": "4", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_subscribe_interval": "86400.000000", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_debug": "false", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_max_payload_keys": "2000", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_max_payload_size": "1048576", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_provider_kill_at": "0", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_requester_kill_at": "0", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_sync_timeout": "60.000000", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_target_pg_per_osd": "100", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_tick_interval": "5", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_timecheck_interval": "300.000000", 2024-01-11T10:08:31.529 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_use_min_delay_socket": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_older_version_delay": "604800", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_filestore_osds": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_misplaced": "false", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_older_version": "true", 2024-01-11T10:08:31.530 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_pool_no_app": "true", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_on_too_few_osds": "false", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "monmap": "", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-01-11T10:08:31.531 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "motr_tracing_enabled": "false", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_op_threads": "3", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_buffer_size": "131072", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_cm": "false", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_device_name": "", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_dscp": "96", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_gid_idx": "0", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_local_gid": "", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_polling_us": "1000", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_port_num": "1", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-01-11T10:08:31.532 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_roce_ver": "1", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_send_buffers": "1024", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_sl": "3", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_support_srq": "true", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_rdma_type": "ib", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_async_reap_threshold": "5", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_before_connect": "false", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_ipv4": "true", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_ipv6": "false", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_msgr1": "true", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_msgr2": "true", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_port_max": "7568", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_port_min": "6800", 2024-01-11T10:08:31.533 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_prefer_ipv4": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_retry_count": "3", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_bind_retry_delay": "5", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_client": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_mds": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_mgr": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_mon": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_blackhole_osd": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_client_mode": "crc", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_cluster_mode": "crc", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_cluster_type": "", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_compress_secure": "false", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_connection_idle_timeout": "900", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_connection_ready_timeout": "10", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_crc_data": "true", 2024-01-11T10:08:31.534 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_crc_header": "true", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_bad_msg": "false", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_bug": "true", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_old_message": "true", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_skipped_message": "false", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_die_on_unhandled_msg": "false", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_coremask": "0xF", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_devs_allowlist": "", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_enable_tso": "true", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_hugepages": "", 2024-01-11T10:08:31.535 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_hw_flow_control": "true", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_lro": "true", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_memory_channel": "4", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_pmd": "", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_port_id": "0", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dump_corrupt_message_level": "1", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_dump_on_send": "false", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_initial_backoff": "0.200000", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_delay_max": "1.000000", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_delay_probability": "0.000000", 2024-01-11T10:08:31.536 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_delay_type": "", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_internal_delays": "0.000000", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_network_congestion": "0", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_inject_socket_failures": "0", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_learn_addr_from_peer": "true", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_max_accept_failures": "4", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_max_backoff": "15.000000", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_mon_client_mode": "crc", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_mon_cluster_mode": "crc", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_mon_service_mode": "crc", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_osd_compress_min_size": "1024", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_osd_compress_mode": "none", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_osd_compression_algorithm": "snappy", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_pq_min_cost": "65536", 2024-01-11T10:08:31.537 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_public_type": "", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_service_mode": "crc", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_listen_backlog": "512", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_nodelay": "true", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_tcp_rcvbuf": "0", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "ms_type": "async+posix", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "no_config_file": "false", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_completion_locks_per_session": "32", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_debug_inject_relock_delay": "false", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_inflight_op_bytes": "104857600", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_inflight_ops": "1024", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_inject_no_watch_ping": "false", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-01-11T10:08:31.538 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_tick_interval": "5.000000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "objecter_timeout": "10.000000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "objectstore_blackhole": "false", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "openssl_engine_opts": "", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_delay_time": "5.000000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_hist_halflife": "1000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_max_low_ops": "2", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_max_ops": "4", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_quantize_effort": "0.100000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_agent_slop": "0.020000", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_allow_recovery_below_min_size": "true", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_asio_thread_count": "2", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_async_recovery_min_cost": "100", 2024-01-11T10:08:31.539 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_auto_mark_unfound_lost": "false", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backfill_retry_interval": "30.000000", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backfill_scan_max": "512", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backfill_scan_min": "64", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backoff_on_degraded": "false", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backoff_on_peering": "false", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_backoff_on_unfound": "true", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_beacon_report_interval": "300", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_duration": "30", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_max_block_size": "67108864", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_bench_small_size_max_iops": "100", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_blkin_trace_all": "false", 2024-01-11T10:08:31.540 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_blocked_scrub_grace_period": "120", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_check_for_log_corruption": "false", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_class_update_on_start": "true", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_message_cap": "256", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_message_size_cap": "524288000", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_op_priority": "63", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_client_watch_timeout": "30", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_command_max_records": "256", 2024-01-11T10:08:31.541 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_compact_on_start": "false", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_chooseleaf_type": "0", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_initial_weight": "-1.000000", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_update_on_start": "true", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_crush_update_weight_set": "true", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_drop_ping_duration": "0", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_feed_pullee": "-1", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-01-11T10:08:31.542 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_misdirected_ops": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_no_acting_change": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_no_purge_strays": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_op_order": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_pg_log_writeout": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_pretend_recovery_active": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_shutdown": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_trim_objects": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_cached_snaps": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_missing_on_start": "false", 2024-01-11T10:08:31.543 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_snaps": "false", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_keys": "1024", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_stride": "524288", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_default_data_pool_replay_window": "5", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_default_notify_timeout": "30", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep": "0.000000", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-01-11T10:08:31.544 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_discard_disconnected_ops": "true", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_enable_op_tracker": "true", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_extblkdev_plugins": "vdo", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_info": "true", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_shutdown": "true", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_fast_shutdown_timeout": "15", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-01-11T10:08:31.545 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_function_tracing": "false", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_grace": "20", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_interval": "6", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_min_peers": "10", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_min_size": "2000", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_stale": "600", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_hit_set_max_size": "100000", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_hit_set_min_size": "1000", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-01-11T10:08:31.546 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_journal_flush_on_shutdown": "true", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_journal_size": "5120", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_kill_backfill_at": "0", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_loop_before_reset_tphandle": "64", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_cache_size": "50", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_dedup": "true", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_message_max": "40", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_message_max_bytes": "10485760", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_map_share_max_epochs": "40", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_attr_name_len": "100", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_attr_size": "0", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_backfills": "1", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_markdown_count": "5", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_markdown_period": "600", 2024-01-11T10:08:31.547 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_object_name_len": "2048", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_object_namespace_len": "256", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_object_size": "134217728", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_omap_entries_per_request": "1024", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pg_blocked_by": "16", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pg_log_entries": "10000", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_pgls": "1024", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_push_cost": "8388608", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_push_objects": "10", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_scrubs": "3", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-01-11T10:08:31.548 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_trimming_pgs": "2", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_write_op_reply_len": "64", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_max_write_size": "90", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_override_recovery_settings": "false", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_profile": "balanced", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-01-11T10:08:31.549 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mclock_skip_benchmark": "false", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_base": "805306368", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_cache_min": "134217728", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_target": "4294967296", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_target_autotune": "false", 2024-01-11T10:08:31.550 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_min_pg_log_entries": "250", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_min_recovery_priority": "0", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_heartbeat_interval": "30", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_report_interval": "5", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_report_max_in_flight": "2", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_num_cache_shards": "32", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_num_op_tracker_shard": "32", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_numa_auto_affinity": "true", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_numa_node": "-1", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_numa_prefer_iface": "true", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-01-11T10:08:31.551 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objecter_finishers": "1", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objectstore": "bluestore", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objectstore_fuse": "false", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_objectstore_tracing": "false", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_complaint_time": "30.000000", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_duration": "600", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_size": "20", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_slow_op_size": "20", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_log_threshold": "5", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_shards": "0", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_shards_hdd": "5", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_shards_ssd": "8", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_threads_per_shard": "0", 2024-01-11T10:08:31.552 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_pq_min_cost": "65536", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_queue": "mclock_scheduler", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_queue_cut_off": "high", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_thread_suicide_timeout": "150", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_op_thread_timeout": "15", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_open_classes_on_start": "true", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_os_flags": "0", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_peering_op_priority": "255", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_delete_cost": "1048576", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_delete_priority": "5", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-01-11T10:08:31.553 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_log_dups_tracked": "3000", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_log_trim_max": "10000", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_log_trim_min": "100", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_object_context_cache_count": "64", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pg_stat_report_interval_max": "500", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_crimson": "true", 2024-01-11T10:08:31.554 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_crush_rule": "-1", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_ec_fast_read": "false", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_bulk": "false", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_nodelete": "false", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_flags": "0", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_min_size": "2", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_pg_num": "32", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_pgp_num": "0", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-01-11T10:08:31.555 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_size": "3", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_default_type": "replicated", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_pool_use_gmt_hitset": "true", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_push_per_object_cost": "1000", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_read_ec_check_for_errors": "false", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recover_clone_overlap": "true", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recover_clone_overlap_limit": "10", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_cost": "20971520", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_delay_start": "0.000000", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_active": "0", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_active_hdd": "3", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_active_ssd": "10", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_chunk": "8388608", 2024-01-11T10:08:31.556 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_max_single_start": "1", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_op_priority": "3", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_op_warn_multiple": "16", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_priority": "5", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_retry_interval": "30.000000", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep": "0.000000", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_repair_during_recovery": "false", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_requested_scrub_priority": "120", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_rollback_to_cluster_snap": "", 2024-01-11T10:08:31.557 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_auto_repair": "false", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_begin_hour": "0", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_begin_week_day": "0", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_chunk_max": "25", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_chunk_min": "5", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_cost": "52428800", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_during_recovery": "false", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_end_hour": "0", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_end_week_day": "0", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_event_cost": "4096", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-01-11T10:08:31.558 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_invalid_stats": "true", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_load_threshold": "0.500000", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_max_interval": "604800.000000", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_max_preemptions": "5", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_min_interval": "86400.000000", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_priority": "5", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_reservation_timeout": "300000", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_sleep": "0.000000", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_shutdown_pgref_assert": "false", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_skip_data_digest": "false", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_smart_report_timeout": "5", 2024-01-11T10:08:31.559 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_cost": "1048576", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_priority": "5", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep": "0.000000", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_stats_update_period_scrubbing": "15", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_target_transaction_size": "30", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-01-11T10:08:31.560 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_default_cache_mode": "writeback", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_tracing": "false", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_use_stale_snap": "false", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "osdc_blkin_trace_all": "false", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_kill_at": "0", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_max_join_drift": "10", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_min": "500", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_min_wait": "0.050000", 2024-01-11T10:08:31.561 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_propose_interval": "1.000000", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_service_trim_max": "500", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_service_trim_max_multiplier": "20", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_service_trim_min": "10", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_stash_full_interval": "25", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_trim_max": "500", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "paxos_trim_min": "250", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "perf": "true", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addr": "-", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "public_addrv": "", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "public_bind_addr": "-", 2024-01-11T10:08:31.562 INFO:teuthology.orchestra.run.smithi040.stdout: "public_network": "", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "public_network_interface": "", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "qat_compressor_enabled": "false", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "qat_compressor_session_max_number": "256", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rados_mon_op_timeout": "0", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rados_osd_op_timeout": "0", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rados_tracing": "false", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_atime_update_interval": "60", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_balance_parent_reads": "false", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_balance_snap_reads": "false", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_blkin_trace_all": "false", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_blocklist_expire_seconds": "0", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_blocklist_on_break_lock": "true", 2024-01-11T10:08:31.563 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache": "true", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_block_writes_upfront": "false", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_max_dirty": "25165824", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_max_dirty_object": "0", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_policy": "writearound", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_size": "33554432", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_target_dirty": "16777216", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_clone_copy_on_read": "false", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_compression_hint": "none", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_concurrent_management_ops": "10", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_clone_format": "auto", 2024-01-11T10:08:31.564 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_data_pool": "", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_features": "61", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_format": "2", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_map_options": "", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_order": "22", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_pool": "rbd", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_stripe_count": "0", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_default_stripe_unit": "0", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_disable_zero_copy_writes": "true", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_discard_granularity_bytes": "65536", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_enable_alloc_hint": "true", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-01-11T10:08:31.565 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_io_scheduler": "simple", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_commit_age": "5.000000", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_flush_interval": "0", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_order": "24", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_pool": "", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_journal_splay_width": "4", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_localize_parent_reads": "false", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_localize_snap_reads": "false", 2024-01-11T10:08:31.566 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_type": "simple", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-01-11T10:08:31.567 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_autotune": "true", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_base": "805306368", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_memory_target": "4294967296", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_delete_delay": "0", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_replay_delay": "0", 2024-01-11T10:08:31.568 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_move_to_trash_on_remove": "false", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_mtime_update_interval": "60", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_non_blocking_aio": "true", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_op_thread_timeout": "60", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_op_threads": "1", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_parent_cache_enabled": "false", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_persistent_cache_mode": "disabled", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_persistent_cache_path": "/tmp", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_persistent_cache_size": "1073741824", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_plugins": "", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_bps_burst": "0", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-01-11T10:08:31.569 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_bps_limit": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_exclude_ops": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_iops_burst": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_iops_limit": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_bps_burst": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_bps_limit": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_iops_burst": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_read_iops_limit": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_schedule_tick_min": "50", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_bps_burst": "0", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-01-11T10:08:31.570 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_bps_limit": "0", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_iops_burst": "0", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_qos_write_iops_limit": "0", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_quiesce_notification_attempts": "10", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_read_from_replica_policy": "default", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_readahead_max_bytes": "524288", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_readahead_trigger_requests": "10", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_request_timed_out_seconds": "30", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_skip_partial_discard": "true", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_tracing": "false", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_validate_names": "true", 2024-01-11T10:08:31.571 INFO:teuthology.orchestra.run.smithi040.stdout: "rbd_validate_pool": "true", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "restapi_base_url": "", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "restapi_log_level": "", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_acl_grants_max_num": "100", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_admin_entry": "admin", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_backend_store": "rados", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_barbican_url": "", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_beast_enable_async": "true", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_counters_cache": "false", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-01-11T10:08:31.572 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_index_max_aio": "128", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_quota_ttl": "600", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cache_enabled": "true", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cache_expiry_interval": "900", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cache_lru_size": "10000", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_config_store": "rados", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_content_length_compat": "false", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_copy_obj_progress": "true", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cors_rules_max_num": "100", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_cross_domain_policy": "", 2024-01-11T10:08:31.573 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_default_encryption_key": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_addr": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_ca_path": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_client_cert": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_client_key": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_password": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_kmip_username": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_require_ssl": "true", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-01-11T10:08:31.574 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_suppress_logs": "true", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_addr": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_auth": "token", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_namespace": "", 2024-01-11T10:08:31.575 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_prefix": "", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_token_file": "", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_buffersize": "524288", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_low_speed_limit": "1024", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_low_speed_time": "300", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_tcp_keepalive": "0", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-01-11T10:08:31.576 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_fadvise": "4", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_host": "127.0.0.1", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_d4n_port": "6379", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_changes_size": "1000", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_num_shards": "128", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_log_window": "30", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_notify_interval_msec": "0", 2024-01-11T10:08:31.577 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_sync_poll_interval": "20", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_data_sync_spawn_window": "20", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_data_log_backing": "fifo", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_region_info_oid": "default.region", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_defer_to_bucket_acls": "", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-01-11T10:08:31.578 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_data_res": "500.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dns_name": "", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dns_s3website_name": "", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_dynamic_resharding": "true", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_gc_threads": "true", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_lc_threads": "true", 2024-01-11T10:08:31.579 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_ops_log": "false", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_quota_threads": "true", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_static_website": "false", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enable_usage_log": "false", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_enforce_swift_acls": "true", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_exit_timeout_secs": "120", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_expose_bucket": "false", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_extended_http_attrs": "", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_filter": "none", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_frontends": "beast port=7480", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_concurrent_io": "10", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_deferred": "50", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-01-11T10:08:31.580 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_objs": "32", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_queue_size": "134213632", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_max_trim_chunk": "16", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_obj_min_wait": "7200", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_processor_max_time": "3600", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_gc_processor_period": "3600", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_get_obj_window_size": "16777216", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_healthcheck_disabling_path": "", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ignore_get_invalid_range": "false", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_init_timeout": "300", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_kafka_connection_idle": "30", 2024-01-11T10:08:31.581 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_kafka_sleep_timeout": "10", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_domain": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_password": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_password_path": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_project": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_tenant": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_token": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_token_path": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_admin_user": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_api_version": "2", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_domain": "", 2024-01-11T10:08:31.582 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_password": "", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_project": "", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_tenant": "", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_barbican_user": "", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_implicit_tenants": "false", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_service_token_enabled": "false", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_token_cache_size": "10000", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_url": "", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_keystone_verify_ssl": "true", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_debug_interval": "-1", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_lock_max_time": "90", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_objs": "32", 2024-01-11T10:08:31.583 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_rules": "1000", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_worker": "3", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_max_wp_worker": "3", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lc_thread_delay": "0", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_dnattr": "uid", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_searchfilter": "", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ldap_uri": "ldaps://", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_http_headers": "", 2024-01-11T10:08:31.584 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_nonexistent_bucket": "false", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_log_object_name_utc": "false", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_attr_name_len": "0", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_attr_size": "0", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_attrs_num_in_req": "0", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_chunk_size": "4194304", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_concurrent_requests": "1024", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_dynamic_shards": "1999", 2024-01-11T10:08:31.585 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_listing_results": "1000", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_notify_retries": "10", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_objs_per_shard": "100000", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_put_param_size": "1048576", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_put_size": "5368709120", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_max_slo_entries": "1000", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_md_log_max_shards": "64", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_md_notify_interval_msec": "200", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_meta_sync_poll_interval": "20", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_meta_sync_spawn_window": "20", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_mp_lock_max_time": "600", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-01-11T10:08:31.586 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_multipart_min_part_size": "5242880", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_fhcache_size": "2017", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_lru_lanes": "5", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_max_gc": "300", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_gc_threads": "false", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_lc_threads": "false", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_quota_threads": "false", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_run_sync_thread": "false", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-01-11T10:08:31.587 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_num_async_rados_threads": "32", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_num_control_oids": "8", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_numa_node": "-1", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_obj_stripe_size": "4194304", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_objexp_chunk_size": "100", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_objexp_gc_interval": "600", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_objexp_hints_num_shards": "127", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_op_thread_timeout": "600", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_op_tracing": "false", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_opa_token": "", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_opa_url": "", 2024-01-11T10:08:31.588 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_opa_verify_ssl": "true", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_rados": "false", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_ops_log_socket_path": "", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_push_interval": "2.000000", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_push_interval_max": "30.000000", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_period_root_pool": ".rgw.root", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-01-11T10:08:31.589 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_lanes": "3", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_max_buckets": "100", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_cache_partitions": "3", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_print_continue": "true", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_print_prohibited_content_length": "false", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rados_tracing": "false", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_realm": "", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_realm_id": "", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-01-11T10:08:31.590 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_region": "", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_region_root_pool": ".rgw.root", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_relaxed_region_enforcement": "false", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_request_uri": "", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_batch_size": "64", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_max_aio": "128", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_num_logs": "16", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_reshard_thread_interval": "600", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_resolve_cname": "false", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_rest_getusage_op_compat": "false", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_run_sync_thread": "true", 2024-01-11T10:08:31.591 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_keystone": "false", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_ldap": "false", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_rados": "true", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_auth_use_sts": "false", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_s3_success_create_obj_status": "0", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_scheduler_type": "throttler", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_script_uri": "", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_service_provider_name": "", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_client_id": "", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_client_secret": "", 2024-01-11T10:08:31.592 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_entry": "sts", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_key": "sts", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_max_session_duration": "43200", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_min_session_duration": "900", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sts_token_introspection_url": "", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_account_in_url": "false", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_auth_entry": "auth", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_auth_url": "", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_custom_header": "", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_enforce_content_length": "false", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_need_stats": "true", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_tenant_name": "", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_token_expiration": "86400", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_url": "", 2024-01-11T10:08:31.593 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_url_prefix": "swift", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_swift_versioning_enabled": "false", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_lease_period": "120", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_interval": "1200", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_obj_etag_verify": "false", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_trace_history_size": "4096", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_thread_pool_size": "512", 2024-01-11T10:08:31.594 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_persistency_max_retries": "0", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_comment": "", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_createby": "", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_encoding": "", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_flag": "false", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_max_size": "5368709120", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_origin": "", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_sha_unit": "524288", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_torrent_tracker": "", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_trust_forwarded_https": "false", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_log_tick_interval": "30", 2024-01-11T10:08:31.595 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_max_shards": "32", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_usage_max_user_shards": "1", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_use_opa_authz": "false", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_counters_cache": "false", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_counters_cache_size": "10000", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_default_quota_max_size": "-1", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_max_buckets": "1000", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_policies_max_num": "100", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_sync_interval": "86400", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_user_unique_email": "true", 2024-01-11T10:08:31.596 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_verify_ssl": "true", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_website_routing_rules_max_num": "50", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zone": "", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zone_id": "", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zonegroup": "", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zonegroup_id": "", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgwlc_auto_session_clear": "true", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rgwlc_skip_bucket_step": "false", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_block_size": "4096", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-01-11T10:08:31.597 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_shard_bits": "4", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_size": "536870912", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cache_type": "binned_lru", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_collect_compaction_stats": "false", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_collect_extended_stats": "false", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_collect_memory_stats": "false", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_index_type": "binary_search", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_log_to_ceph_log": "true", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_metadata_block_size": "4096", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_partition_filters": "false", 2024-01-11T10:08:31.598 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_perf": "false", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "rotating_keys_renewal_timeout": "10", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "run_dir": "/var/run/ceph", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_block_create": "true", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_cache_lru_size": "67108864", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_cbjournal_size": "5368709120", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_data_delta_based_overwrite": "0", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_default_max_object_size": "16777216", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_device_size": "53687091200", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_batch_capacity": "16", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-01-11T10:08:31.599 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_journal_iodepth_limit": "5", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_main_device_type": "SSD", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_max_concurrent_transactions": "8", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "seastore_segment_size": "67108864", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "set_keepcaps": "false", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "setgroup": "", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "setuser": "", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "setuser_match_path": "", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "target_max_misplaced_ratio": "0.050000", 2024-01-11T10:08:31.600 INFO:teuthology.orchestra.run.smithi040.stdout: "thp": "false", 2024-01-11T10:08:31.601 INFO:teuthology.orchestra.run.smithi040.stdout: "threadpool_default_timeout": "60", 2024-01-11T10:08:31.601 INFO:teuthology.orchestra.run.smithi040.stdout: "threadpool_empty_queue_max_wait": "2", 2024-01-11T10:08:31.601 INFO:teuthology.orchestra.run.smithi040.stdout: "throttler_perf_counter": "true" 2024-01-11T10:08:31.601 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-01-11T10:08:31.603 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-01-11T10:08:31.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.720+0000 7fb68379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.710 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.724+0000 7fb68379e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.723 INFO:teuthology.orchestra.run.smithi040.stdout:{} 2024-01-11T10:08:31.723 INFO:teuthology.orchestra.run.smithi040.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-01-11T10:08:31.733 DEBUG:teuthology.orchestra.run.smithi040:> type ceph-objectstore-tool 2024-01-11T10:08:31.746 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-01-11T10:08:31.747 DEBUG:teuthology.orchestra.run.smithi134:> type ceph-objectstore-tool 2024-01-11T10:08:31.764 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-01-11T10:08:31.764 INFO:teuthology.run_tasks:Running task full_sequential... 2024-01-11T10:08:31.773 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-01-11T10:08:31.775 INFO:tasks.radosbench:Beginning radosbench... 2024-01-11T10:08:31.775 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-01-11T10:08:31.775 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-01-11T10:08:31.775 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-01-11T10:08:31.776 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:08:31.776 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:08:31.776 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:08:31.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:08:31.776 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-01-11T10:08:31.776 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.909+0000 7f3dbbf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.913+0000 7f3dbbf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.909+0000 7faf9d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.909+0000 7f923a2f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.917+0000 7faf9d06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:31.920+0000 7f923a2f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:31.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:31.923 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:31.923 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:08:32.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:32.043+0000 7f82e2d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:32.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:32.051+0000 7f82e2d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:32.050 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:32.051 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:32.051 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:32.053 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:32.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:08:32.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:32.171+0000 7f849f51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:32.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:32.179+0000 7f849f51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:32.179 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:32.180 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:32.182 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:32.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:08:32.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:32.300+0000 7f3f3e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:32.292 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:32.307+0000 7f3f3e42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:32.306 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:32.307 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:32.309 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:32.938 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_0' created 2024-01-11T10:08:32.957 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-01-11T10:08:33.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:33.088+0000 7f50cbf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:33.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:33.099+0000 7f50cbf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:33.939 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-01-11T10:08:33.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:08:34.099 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:34.112+0000 7fa8bb31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:34.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:34.118+0000 7fa8bb31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:34.958 DEBUG:teuthology.orchestra.run.smithi040:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-01-11T10:08:34.961 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-01-11T10:08:34.961 INFO:teuthology.orchestra.run:waiting for 3000 2024-01-11T10:08:35.021 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:hints = 1 2024-01-11T10:08:35.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-01-11T10:08:35.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object prefix: benchmark_data_smithi040_30941 2024-01-11T10:08:35.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:08:35.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 0 0 0 0 0 0 - 0 2024-01-11T10:08:36.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1 16 213 197 12.3116 12.3125 0.0403073 0.0768078 2024-01-11T10:08:36.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:36.405 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:36.419+0000 7f9a3b217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:36.409 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:36.425+0000 7f9a3b217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2 16 435 419 13.0925 13.875 0.0381261 0.0747446 2024-01-11T10:08:37.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:08:37.310 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:08:37.311 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:08:37.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:08:37.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.452+0000 7fe71aa17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.461+0000 7fe71aa17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.464 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:37.465 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:37.468 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:37.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:08:37.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.604+0000 7f0183a06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.598 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.613+0000 7f0183a06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:37.635 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:37.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:37.638 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:37.638 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:08:37.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.782+0000 7f10f839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.791+0000 7f10f839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:37.791 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:37.792 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:37.795 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:37.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:08:37.923 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.933+0000 7f8db18f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:37.942+0000 7f8db18f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:37.967 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:37.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:37.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:37.971 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:38.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3 16 646 630 13.1239 13.1875 0.0305938 0.0750251 2024-01-11T10:08:38.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:08:38.713 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:38.727+0000 7f759c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:38.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:38.733+0000 7f759c06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:39.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 4 16 861 845 13.202 13.4375 0.0669108 0.0745986 2024-01-11T10:08:40.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 5 16 1083 1067 13.3364 13.875 0.072397 0.0742603 2024-01-11T10:08:40.939 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:41.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 6 16 1314 1298 13.5198 14.4375 0.0340294 0.0734797 2024-01-11T10:08:41.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:41.069+0000 7f5f4e99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:41.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:41.074+0000 7f5f4e99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:42.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 7 16 1548 1532 13.6775 14.625 0.102573 0.0726948 2024-01-11T10:08:42.972 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:08:42.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:08:42.972 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:08:43.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 8 16 1772 1756 13.7177 14 0.0504906 0.0725597 2024-01-11T10:08:43.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:43.117+0000 7f2fea4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:43.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:43.125+0000 7f2fea4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:43.277 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:08:43.411 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:43.412 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:08:43.414 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:43.428+0000 7efcf662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:43.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:43.434+0000 7efcf662d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:43.432 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-01-11T10:08:42.102494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":36438560,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":684,"num_write_kb":36902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":628,"ondisk_log_size":628,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":112572,"kb_used_data":5268,"kb_used_omap":0,"kb_used_meta":107264,"kb_avail":377374788,"statfs":{"total":386547056640,"available":386431782912,"internally_reserved":0,"allocated":5394432,"data_stored":76122676,"data_compressed":3611223,"data_compressed_allocated":4759552,"data_compressed_original":75907072,"omap_allocated":0,"internal_metadata":109838336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":35848192,"num_objects":547,"num_object_clones":0,"num_object_copies":1641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":35008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.146982"},"pg_stats":[{"pgid":"3.f","version":"17'46","reported_seq":58,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:37.883535+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:08:37.883535+0000","last_peered":"2024-01-11T10:08:37.883535+0000","last_clean":"2024-01-11T10:08:37.883535+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:08:37.883535+0000","last_undegraded":"2024-01-11T10:08:37.883535+0000","last_fullsized":"2024-01-11T10:08:37.883535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'42","reported_seq":54,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:37.929291+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:08:37.929291+0000","last_peered":"2024-01-11T10:08:37.929291+0000","last_clean":"2024-01-11T10:08:37.929291+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:08:37.929291+0000","last_undegraded":"2024-01-11T10:08:37.929291+0000","last_fullsized":"2024-01-11T10:08:37.929291+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'24","reported_seq":36,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:36.746758+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:08:36.746758+0000","last_peered":"2024-01-11T10:08:36.746758+0000","last_clean":"2024-01-11T10:08:36.746758+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:08:36.746758+0000","last_undegraded":"2024-01-11T10:08:36.746758+0000","last_fullsized":"2024-01-11T10:08:36.746758+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":1600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'55","reported_seq":67,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:38.078505+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:08:38.078505+0000","last_peered":"2024-01-11T10:08:38.078505+0000","last_clean":"2024-01-11T10:08:38.078505+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:08:38.078505+0000","last_undegraded":"2024-01-11T10:08:38.078505+0000","last_fullsized":"2024-01-11T10:08:38.078505+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":56,"num_write_kb":3584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'17","reported_seq":29,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:36.627949+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:08:36.627949+0000","last_peered":"2024-01-11T10:08:36.627949+0000","last_clean":"2024-01-11T10:08:36.627949+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:08:36.627949+0000","last_undegraded":"2024-01-11T10:08:36.627949+0000","last_fullsized":"2024-01-11T10:08:36.627949+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":1152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'32","reported_seq":44,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:38.017341+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:08:38.017341+0000","last_peered":"2024-01-11T10:08:38.017341+0000","last_clean":"2024-01-11T10:08:38.017341+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:08:38.017341+0000","last_undegraded":"2024-01-11T10:08:38.017341+0000","last_fullsized":"2024-01-11T10:08:38.017341+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'34","reported_seq":46,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:38.099284+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:08:38.099284+0000","last_peered":"2024-01-11T10:08:38.099284+0000","last_clean":"2024-01-11T10:08:38.099284+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:08:38.099284+0000","last_undegraded":"2024-01-11T10:08:38.099284+0000","last_fullsized":"2024-01-11T10:08:38.099284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2293760,"num_objects":35,"num_object_clones":0,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35,"num_write_kb":2240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'20","reported_seq":32,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:37.091085+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:08:37.091085+0000","last_peered":"2024-01-11T10:08:37.091085+0000","last_clean":"2024-01-11T10:08:37.091085+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:08:37.091085+0000","last_undegraded":"2024-01-11T10:08:37.091085+0000","last_fullsized":"2024-01-11T10:08:37.091085+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1376256,"num_objects":21,"num_object_clones":0,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":1344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.992304+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:34.992304+0000","last_peered":"2024-01-11T10:08:34.992304+0000","last_clean":"2024-01-11T10:08:34.992304+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:34.992304+0000","last_undegraded":"2024-01-11T10:08:34.992304+0000","last_fullsized":"2024-01-11T10:08:34.992304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'26","reported_seq":38,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:36.988300+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:08:36.988300+0000","last_peered":"2024-01-11T10:08:36.988300+0000","last_clean":"2024-01-11T10:08:36.988300+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:08:36.988300+0000","last_undegraded":"2024-01-11T10:08:36.988300+0000","last_fullsized":"2024-01-11T10:08:36.988300+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'37","reported_seq":49,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:37.048537+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:08:37.048537+0000","last_peered":"2024-01-11T10:08:37.048537+0000","last_clean":"2024-01-11T10:08:37.048537+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:08:37.048537+0000","last_undegraded":"2024-01-11T10:08:37.048537+0000","last_fullsized":"2024-01-11T10:08:37.048537+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2490368,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":2432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.976416+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:34.976416+0000","last_peered":"2024-01-11T10:08:34.976416+0000","last_clean":"2024-01-11T10:08:34.976416+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:34.976416+0000","last_undegraded":"2024-01-11T10:08:34.976416+0000","last_fullsized":"2024-01-11T10:08:34.976416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'26","reported_seq":38,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:37.061746+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:08:37.061746+0000","last_peered":"2024-01-11T10:08:37.061746+0000","last_clean":"2024-01-11T10:08:37.061746+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:08:37.061746+0000","last_undegraded":"2024-01-11T10:08:37.061746+0000","last_fullsized":"2024-01-11T10:08:37.061746+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":1728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052675+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052675+0000","last_peered":"2024-01-11T10:08:35.052675+0000","last_clean":"2024-01-11T10:08:35.052675+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052675+0000","last_undegraded":"2024-01-11T10:08:35.052675+0000","last_fullsized":"2024-01-11T10:08:35.052675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'38","reported_seq":50,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:38.052652+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:08:38.052652+0000","last_peered":"2024-01-11T10:08:38.052652+0000","last_clean":"2024-01-11T10:08:38.052652+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:08:38.052652+0000","last_undegraded":"2024-01-11T10:08:38.052652+0000","last_fullsized":"2024-01-11T10:08:38.052652+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39,"num_write_kb":2496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'29","reported_seq":41,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:36.955798+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:08:36.955798+0000","last_peered":"2024-01-11T10:08:36.955798+0000","last_clean":"2024-01-11T10:08:36.955798+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:08:36.955798+0000","last_undegraded":"2024-01-11T10:08:36.955798+0000","last_fullsized":"2024-01-11T10:08:36.955798+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.073743+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:35.073743+0000","last_peered":"2024-01-11T10:08:35.073743+0000","last_clean":"2024-01-11T10:08:35.073743+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:35.073743+0000","last_undegraded":"2024-01-11T10:08:35.073743+0000","last_fullsized":"2024-01-11T10:08:35.073743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'32","reported_seq":44,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:38.023591+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:08:38.023591+0000","last_peered":"2024-01-11T10:08:38.023591+0000","last_clean":"2024-01-11T10:08:38.023591+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:08:38.023591+0000","last_undegraded":"2024-01-11T10:08:38.023591+0000","last_fullsized":"2024-01-11T10:08:38.023591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2162688,"num_objects":33,"num_object_clones":0,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33,"num_write_kb":2112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'29","reported_seq":41,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:36.773996+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:08:36.773996+0000","last_peered":"2024-01-11T10:08:36.773996+0000","last_clean":"2024-01-11T10:08:36.773996+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:08:36.773996+0000","last_undegraded":"2024-01-11T10:08:36.773996+0000","last_fullsized":"2024-01-11T10:08:36.773996+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1966080,"num_objects":30,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30,"num_write_kb":1920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.071023+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:35.071023+0000","last_peered":"2024-01-11T10:08:35.071023+0000","last_clean":"2024-01-11T10:08:35.071023+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:35.071023+0000","last_undegraded":"2024-01-11T10:08:35.071023+0000","last_fullsized":"2024-01-11T10:08:35.071023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'44","reported_seq":56,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:38.008721+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:08:38.008721+0000","last_peered":"2024-01-11T10:08:38.008721+0000","last_clean":"2024-01-11T10:08:38.008721+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:08:38.008721+0000","last_undegraded":"2024-01-11T10:08:38.008721+0000","last_fullsized":"2024-01-11T10:08:38.008721+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.984172+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:34.984172+0000","last_peered":"2024-01-11T10:08:34.984172+0000","last_clean":"2024-01-11T10:08:34.984172+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:34.984172+0000","last_undegraded":"2024-01-11T10:08:34.984172+0000","last_fullsized":"2024-01-11T10:08:34.984172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":35848192,"num_objects":547,"num_object_clones":0,"num_object_copies":1641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":547,"num_write_kb":35008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":531,"ondisk_log_size":531,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28752,"kb_used_data":1924,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94343088,"statfs":{"total":96636764160,"available":96607322112,"internally_reserved":0,"allocated":1970176,"data_stored":28877589,"data_compressed":1370742,"data_compressed_allocated":1806336,"data_compressed_original":28819456,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28268,"kb_used_data":1444,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94343572,"statfs":{"total":96636764160,"available":96607817728,"internally_reserved":0,"allocated":1478656,"data_stored":21013269,"data_compressed":997409,"data_compressed_allocated":1314816,"data_compressed_original":20955136,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705683,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27856,"kb_used_data":1028,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":1052672,"data_stored":14197525,"data_compressed":673845,"data_compressed_allocated":888832,"data_compressed_original":14139392,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705684,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27696,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344144,"statfs":{"total":96636764160,"available":96608403456,"internally_reserved":0,"allocated":892928,"data_stored":12034293,"data_compressed":569227,"data_compressed_allocated":749568,"data_compressed_original":11993088,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:08:44.022 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 9 16 2011 1995 13.8531 14.9375 0.034153 0.0718551 2024-01-11T10:08:45.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 10 16 2250 2234 13.9614 14.9375 0.0939681 0.0713318 2024-01-11T10:08:45.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:45.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:45.735+0000 7f88a20f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:45.724 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:45.740+0000 7f88a20f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:46.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 11 16 2461 2445 13.8909 13.1875 0.0948429 0.0717471 2024-01-11T10:08:47.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 12 16 2674 2658 13.8426 13.3125 0.114648 0.0716649 2024-01-11T10:08:47.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:08:48.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 13 16 2897 2881 13.8498 13.9375 0.0707195 0.0719628 2024-01-11T10:08:48.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:48.060+0000 7f6bbe06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:48.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:48.066+0000 7f6bbe06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:48.434 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:08:48.435 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:08:48.435 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:08:48.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:48.572+0000 7f58124ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:48.565 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:48.580+0000 7f58124ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:08:48.838 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:08:48.851 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-01-11T10:08:48.103571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":148701728,"num_objects":2262,"num_object_clones":0,"num_object_copies":6786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":2397,"num_write_kb":146534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2341,"ondisk_log_size":2341,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":158336,"kb_used_data":26080,"kb_used_omap":0,"kb_used_meta":132224,"kb_avail":377329024,"statfs":{"total":386547056640,"available":386384920576,"internally_reserved":0,"allocated":26705920,"data_stored":417106484,"data_compressed":19801091,"data_compressed_allocated":26071040,"data_compressed_original":416890880,"omap_allocated":0,"internal_metadata":135397376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2260,"num_write_kb":144640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002184"},"pg_stats":[{"pgid":"3.f","version":"17'122","reported_seq":134,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:43.017861+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:08:43.017861+0000","last_peered":"2024-01-11T10:08:43.017861+0000","last_clean":"2024-01-11T10:08:43.017861+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:08:43.017861+0000","last_undegraded":"2024-01-11T10:08:43.017861+0000","last_fullsized":"2024-01-11T10:08:43.017861+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":123,"num_write_kb":7872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'105","reported_seq":117,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:43.077420+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:08:43.077420+0000","last_peered":"2024-01-11T10:08:43.077420+0000","last_clean":"2024-01-11T10:08:43.077420+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:08:43.077420+0000","last_undegraded":"2024-01-11T10:08:43.077420+0000","last_fullsized":"2024-01-11T10:08:43.077420+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":106,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":6784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'170","reported_seq":182,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:47.073020+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:08:47.073020+0000","last_peered":"2024-01-11T10:08:47.073020+0000","last_clean":"2024-01-11T10:08:47.073020+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:08:47.073020+0000","last_undegraded":"2024-01-11T10:08:47.073020+0000","last_fullsized":"2024-01-11T10:08:47.073020+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":171,"num_write_kb":10944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'121","reported_seq":133,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:43.022549+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:08:43.022549+0000","last_peered":"2024-01-11T10:08:43.022549+0000","last_clean":"2024-01-11T10:08:43.022549+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:08:43.022549+0000","last_undegraded":"2024-01-11T10:08:43.022549+0000","last_fullsized":"2024-01-11T10:08:43.022549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":122,"num_write_kb":7808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'165","reported_seq":177,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:46.980989+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:08:46.980989+0000","last_peered":"2024-01-11T10:08:46.980989+0000","last_clean":"2024-01-11T10:08:46.980989+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:08:46.980989+0000","last_undegraded":"2024-01-11T10:08:46.980989+0000","last_fullsized":"2024-01-11T10:08:46.980989+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":166,"num_write_kb":10624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'103","reported_seq":115,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:42.960757+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:08:42.960757+0000","last_peered":"2024-01-11T10:08:42.960757+0000","last_clean":"2024-01-11T10:08:42.960757+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:08:42.960757+0000","last_undegraded":"2024-01-11T10:08:42.960757+0000","last_fullsized":"2024-01-11T10:08:42.960757+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'97","reported_seq":109,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:43.085842+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:08:43.085842+0000","last_peered":"2024-01-11T10:08:43.085842+0000","last_clean":"2024-01-11T10:08:43.085842+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:08:43.085842+0000","last_undegraded":"2024-01-11T10:08:43.085842+0000","last_fullsized":"2024-01-11T10:08:43.085842+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":98,"num_write_kb":6272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'149","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:46.813356+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:08:46.813356+0000","last_peered":"2024-01-11T10:08:46.813356+0000","last_clean":"2024-01-11T10:08:46.813356+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:08:46.813356+0000","last_undegraded":"2024-01-11T10:08:46.813356+0000","last_fullsized":"2024-01-11T10:08:46.813356+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9830400,"num_objects":150,"num_object_clones":0,"num_object_copies":450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":150,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.992304+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:34.992304+0000","last_peered":"2024-01-11T10:08:34.992304+0000","last_clean":"2024-01-11T10:08:34.992304+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:34.992304+0000","last_undegraded":"2024-01-11T10:08:34.992304+0000","last_fullsized":"2024-01-11T10:08:34.992304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'187","reported_seq":199,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:46.988212+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:08:46.988212+0000","last_peered":"2024-01-11T10:08:46.988212+0000","last_clean":"2024-01-11T10:08:46.988212+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:08:46.988212+0000","last_undegraded":"2024-01-11T10:08:46.988212+0000","last_fullsized":"2024-01-11T10:08:46.988212+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12320768,"num_objects":188,"num_object_clones":0,"num_object_copies":564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":188,"num_write_kb":12032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'176","reported_seq":188,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:47.104321+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:08:47.104321+0000","last_peered":"2024-01-11T10:08:47.104321+0000","last_clean":"2024-01-11T10:08:47.104321+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:08:47.104321+0000","last_undegraded":"2024-01-11T10:08:47.104321+0000","last_fullsized":"2024-01-11T10:08:47.104321+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":177,"num_write_kb":11328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.976416+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:34.976416+0000","last_peered":"2024-01-11T10:08:34.976416+0000","last_clean":"2024-01-11T10:08:34.976416+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:34.976416+0000","last_undegraded":"2024-01-11T10:08:34.976416+0000","last_fullsized":"2024-01-11T10:08:34.976416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'161","reported_seq":173,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:46.963927+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:08:46.963927+0000","last_peered":"2024-01-11T10:08:46.963927+0000","last_clean":"2024-01-11T10:08:46.963927+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:08:46.963927+0000","last_undegraded":"2024-01-11T10:08:46.963927+0000","last_fullsized":"2024-01-11T10:08:46.963927+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10616832,"num_objects":162,"num_object_clones":0,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":162,"num_write_kb":10368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052675+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052675+0000","last_peered":"2024-01-11T10:08:35.052675+0000","last_clean":"2024-01-11T10:08:35.052675+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052675+0000","last_undegraded":"2024-01-11T10:08:35.052675+0000","last_fullsized":"2024-01-11T10:08:35.052675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'120","reported_seq":132,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:43.050764+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:08:43.050764+0000","last_peered":"2024-01-11T10:08:43.050764+0000","last_clean":"2024-01-11T10:08:43.050764+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:08:43.050764+0000","last_undegraded":"2024-01-11T10:08:43.050764+0000","last_fullsized":"2024-01-11T10:08:43.050764+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7929856,"num_objects":121,"num_object_clones":0,"num_object_copies":363,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":121,"num_write_kb":7744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'181","reported_seq":193,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:46.845035+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:08:46.845035+0000","last_peered":"2024-01-11T10:08:46.845035+0000","last_clean":"2024-01-11T10:08:46.845035+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:08:46.845035+0000","last_undegraded":"2024-01-11T10:08:46.845035+0000","last_fullsized":"2024-01-11T10:08:46.845035+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":182,"num_write_kb":11648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.073743+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:35.073743+0000","last_peered":"2024-01-11T10:08:35.073743+0000","last_clean":"2024-01-11T10:08:35.073743+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:35.073743+0000","last_undegraded":"2024-01-11T10:08:35.073743+0000","last_fullsized":"2024-01-11T10:08:35.073743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'102","reported_seq":114,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:43.030863+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:08:43.030863+0000","last_peered":"2024-01-11T10:08:43.030863+0000","last_clean":"2024-01-11T10:08:43.030863+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:08:43.030863+0000","last_undegraded":"2024-01-11T10:08:43.030863+0000","last_fullsized":"2024-01-11T10:08:43.030863+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6750208,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":6592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'177","reported_seq":189,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:46.928492+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:08:46.928492+0000","last_peered":"2024-01-11T10:08:46.928492+0000","last_clean":"2024-01-11T10:08:46.928492+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:08:46.928492+0000","last_undegraded":"2024-01-11T10:08:46.928492+0000","last_fullsized":"2024-01-11T10:08:46.928492+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":178,"num_write_kb":11392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.071023+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:35.071023+0000","last_peered":"2024-01-11T10:08:35.071023+0000","last_clean":"2024-01-11T10:08:35.071023+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:35.071023+0000","last_undegraded":"2024-01-11T10:08:35.071023+0000","last_fullsized":"2024-01-11T10:08:35.071023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'108","reported_seq":120,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:42.970546+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:08:42.970546+0000","last_peered":"2024-01-11T10:08:42.970546+0000","last_clean":"2024-01-11T10:08:42.970546+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:08:42.970546+0000","last_undegraded":"2024-01-11T10:08:42.970546+0000","last_fullsized":"2024-01-11T10:08:42.970546+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109,"num_write_kb":6976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.984172+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:34.984172+0000","last_peered":"2024-01-11T10:08:34.984172+0000","last_clean":"2024-01-11T10:08:34.984172+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:34.984172+0000","last_undegraded":"2024-01-11T10:08:34.984172+0000","last_fullsized":"2024-01-11T10:08:34.984172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":148111360,"num_objects":2260,"num_object_clones":0,"num_object_copies":6780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2260,"num_write_kb":144640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2244,"ondisk_log_size":2244,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":36000,"kb_used_data":5016,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94335840,"statfs":{"total":96636764160,"available":96599900160,"internally_reserved":0,"allocated":5136384,"data_stored":79536917,"data_compressed":3776085,"data_compressed_allocated":4972544,"data_compressed_original":79478784,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705689,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":35556,"kb_used_data":4572,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94336284,"statfs":{"total":96636764160,"available":96600354816,"internally_reserved":0,"allocated":4681728,"data_stored":72262421,"data_compressed":3430615,"data_compressed_allocated":4517888,"data_compressed_original":72204288,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44068,"kb_used_data":8924,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94327772,"statfs":{"total":96636764160,"available":96591638528,"internally_reserved":0,"allocated":9138176,"data_stored":143565589,"data_compressed":6816247,"data_compressed_allocated":8974336,"data_compressed_original":143507456,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705693,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":42712,"kb_used_data":7568,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94329128,"statfs":{"total":96636764160,"available":96593027072,"internally_reserved":0,"allocated":7749632,"data_stored":121741557,"data_compressed":5778144,"data_compressed_allocated":7606272,"data_compressed_original":121700352,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:08:49.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 14 16 3116 3100 13.8381 13.6875 0.0887639 0.0720578 2024-01-11T10:08:50.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 15 16 3345 3329 13.8697 14.3125 0.075037 0.0718647 2024-01-11T10:08:50.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:50.468 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:50.483+0000 7f714dd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:50.472 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:50.488+0000 7f714dd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:51.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 16 16 3560 3544 13.8426 13.4375 0.040185 0.0720571 2024-01-11T10:08:52.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 17 16 3769 3753 13.7967 13.0625 0.0826329 0.0722911 2024-01-11T10:08:52.635 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:08:52.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:52.788+0000 7ff678f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:52.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:52.793+0000 7ff678f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:53.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 18 16 3990 3974 13.7975 13.8125 0.0869766 0.0722516 2024-01-11T10:08:53.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:08:53.853 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:08:53.853 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:08:53.853 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:08:53.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.007+0000 7f5ec6e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.017+0000 7f5ec6e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 19 16 4227 4211 13.8508 14.8125 0.0750668 0.0720783 2024-01-11T10:08:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:54.044 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:54.045 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:54.048 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:54.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:08:54.192 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.203+0000 7f0aa162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.211+0000 7f0aa162d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:54.258 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:54.259 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:54.259 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:54.260 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:54.260 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:54.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:08:54.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.398+0000 7fee9ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.408+0000 7fee9ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:54.416 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:54.417 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:54.419 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:54.419 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:54.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:08:54.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.573+0000 7f09ebe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:54.581+0000 7f09ebe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:54.642 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:54.643 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:54.643 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:54.645 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:54.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:55.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:08:55.039840+0000 min lat: 0.0112618 max lat: 0.19089 avg lat: 0.0721748 2024-01-11T10:08:55.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:08:55.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 20 16 4439 4423 13.8207 13.25 0.123389 0.0721748 2024-01-11T10:08:55.093 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:55.107+0000 7fadc255e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:55.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:55.113+0000 7fadc255e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:56.023 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 21 16 4645 4629 13.7756 12.875 0.0278779 0.072424 2024-01-11T10:08:57.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 22 16 4862 4846 13.7659 13.5625 0.0299318 0.0724957 2024-01-11T10:08:57.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:08:57.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:57.408+0000 7f7d3a19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:57.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:57.413+0000 7f7d3a19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:58.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 23 16 5080 5064 13.7597 13.625 0.0389481 0.0725355 2024-01-11T10:08:59.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 24 16 5287 5271 13.7254 12.9375 0.101287 0.0727349 2024-01-11T10:08:59.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:08:59.646 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:08:59.646 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:08:59.647 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:08:59.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:08:59.752 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:59.766+0000 7fbf6375e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:59.756 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:59.772+0000 7fbf6375e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:59.782 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:59.792+0000 7f4f848f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:59.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:59.801+0000 7f4f848f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:59.802 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:59.805 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:59.805 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:08:59.944 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:59.955+0000 7f798ca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:59.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:08:59.965+0000 7f798ca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:08:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:08:59.967 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:08:59.968 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:08:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:08:59.971 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:08:59.971 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:00.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 25 16 5491 5475 13.6864 12.75 0.0233999 0.0729 2024-01-11T10:09:00.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:00.158+0000 7f309fb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:00.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:00.167+0000 7f309fb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:00.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:00.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:00.193 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:00.193 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:00.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:00.380+0000 7fc9a899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:00.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:00.389+0000 7fc9a899e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:00.420 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:00.421 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:00.424 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:01.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 26 16 5708 5692 13.6815 13.5625 0.109719 0.0729702 2024-01-11T10:09:01.936 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:02.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 27 16 5924 5908 13.6748 13.5 0.0415551 0.0729927 2024-01-11T10:09:02.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:02.097+0000 7f2f5259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:02.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:02.101+0000 7f2f5259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:03.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 28 16 6128 6112 13.6417 12.75 0.0780667 0.0731573 2024-01-11T10:09:04.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 29 16 6348 6332 13.6454 13.75 0.0330845 0.0731975 2024-01-11T10:09:04.279 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:04.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:04.419+0000 7f0e41f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:04.407 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:04.423+0000 7f0e41f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:05.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 30 16 6550 6534 13.6114 12.625 0.0524373 0.0733792 2024-01-11T10:09:05.425 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:05.425 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:05.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:09:05.542 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:05.553+0000 7fb1d7664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:05.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:05.562+0000 7fb1d7664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:05.764 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:09:05.765 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:09:05.779 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-01-11T10:09:04.106078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":397148704,"num_objects":6053,"num_object_clones":0,"num_object_copies":18159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":6188,"num_write_kb":389158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6132,"ondisk_log_size":6132,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":182184,"kb_used_data":70728,"kb_used_omap":0,"kb_used_meta":111424,"kb_avail":377305176,"statfs":{"total":386547056640,"available":386360500224,"internally_reserved":0,"allocated":72425472,"data_stored":1148619316,"data_compressed":54537235,"data_compressed_allocated":71790592,"data_compressed_original":1148403712,"omap_allocated":0,"internal_metadata":114098176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":212271104,"num_objects":3239,"num_object_clones":0,"num_object_copies":9717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3239,"num_write_kb":207296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001819"},"pg_stats":[{"pgid":"3.f","version":"17'381","reported_seq":393,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.050591+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:09:03.050591+0000","last_peered":"2024-01-11T10:09:03.050591+0000","last_clean":"2024-01-11T10:09:03.050591+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:09:03.050591+0000","last_undegraded":"2024-01-11T10:09:03.050591+0000","last_fullsized":"2024-01-11T10:09:03.050591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25034752,"num_objects":382,"num_object_clones":0,"num_object_copies":1146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":382,"num_write_kb":24448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'372","reported_seq":384,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.004731+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:09:03.004731+0000","last_peered":"2024-01-11T10:09:03.004731+0000","last_clean":"2024-01-11T10:09:03.004731+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:09:03.004731+0000","last_undegraded":"2024-01-11T10:09:03.004731+0000","last_fullsized":"2024-01-11T10:09:03.004731+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":373,"num_object_clones":0,"num_object_copies":1119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":373,"num_write_kb":23872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'360","reported_seq":372,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:02.029675+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:09:02.029675+0000","last_peered":"2024-01-11T10:09:02.029675+0000","last_clean":"2024-01-11T10:09:02.029675+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:09:02.029675+0000","last_undegraded":"2024-01-11T10:09:02.029675+0000","last_fullsized":"2024-01-11T10:09:02.029675+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23658496,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":361,"num_write_kb":23104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'370","reported_seq":382,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.092198+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:09:03.092198+0000","last_peered":"2024-01-11T10:09:03.092198+0000","last_clean":"2024-01-11T10:09:03.092198+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:09:03.092198+0000","last_undegraded":"2024-01-11T10:09:03.092198+0000","last_fullsized":"2024-01-11T10:09:03.092198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24313856,"num_objects":371,"num_object_clones":0,"num_object_copies":1113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":371,"num_write_kb":23744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'363","reported_seq":375,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:02.124389+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:09:02.124389+0000","last_peered":"2024-01-11T10:09:02.124389+0000","last_clean":"2024-01-11T10:09:02.124389+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:09:02.124389+0000","last_undegraded":"2024-01-11T10:09:02.124389+0000","last_fullsized":"2024-01-11T10:09:02.124389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23855104,"num_objects":364,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":364,"num_write_kb":23296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'399","reported_seq":411,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.051400+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:09:03.051400+0000","last_peered":"2024-01-11T10:09:03.051400+0000","last_clean":"2024-01-11T10:09:03.051400+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:09:03.051400+0000","last_undegraded":"2024-01-11T10:09:03.051400+0000","last_fullsized":"2024-01-11T10:09:03.051400+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26214400,"num_objects":400,"num_object_clones":0,"num_object_copies":1200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":400,"num_write_kb":25600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'369","reported_seq":381,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.132249+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:09:03.132249+0000","last_peered":"2024-01-11T10:09:03.132249+0000","last_clean":"2024-01-11T10:09:03.132249+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:09:03.132249+0000","last_undegraded":"2024-01-11T10:09:03.132249+0000","last_fullsized":"2024-01-11T10:09:03.132249+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24248320,"num_objects":370,"num_object_clones":0,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":370,"num_write_kb":23680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'351","reported_seq":363,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:01.982131+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:09:01.982131+0000","last_peered":"2024-01-11T10:09:01.982131+0000","last_clean":"2024-01-11T10:09:01.982131+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:09:01.982131+0000","last_undegraded":"2024-01-11T10:09:01.982131+0000","last_fullsized":"2024-01-11T10:09:01.982131+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.992304+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:34.992304+0000","last_peered":"2024-01-11T10:08:34.992304+0000","last_clean":"2024-01-11T10:08:34.992304+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:34.992304+0000","last_undegraded":"2024-01-11T10:08:34.992304+0000","last_fullsized":"2024-01-11T10:08:34.992304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'394","reported_seq":406,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:02.066834+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:09:02.066834+0000","last_peered":"2024-01-11T10:09:02.066834+0000","last_clean":"2024-01-11T10:09:02.066834+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:09:02.066834+0000","last_undegraded":"2024-01-11T10:09:02.066834+0000","last_fullsized":"2024-01-11T10:09:02.066834+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":395,"num_object_clones":0,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":395,"num_write_kb":25280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'389","reported_seq":401,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:01.854478+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:09:01.854478+0000","last_peered":"2024-01-11T10:09:01.854478+0000","last_clean":"2024-01-11T10:09:01.854478+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:09:01.854478+0000","last_undegraded":"2024-01-11T10:09:01.854478+0000","last_fullsized":"2024-01-11T10:09:01.854478+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25559040,"num_objects":390,"num_object_clones":0,"num_object_copies":1170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":390,"num_write_kb":24960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.976416+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:34.976416+0000","last_peered":"2024-01-11T10:08:34.976416+0000","last_clean":"2024-01-11T10:08:34.976416+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:34.976416+0000","last_undegraded":"2024-01-11T10:08:34.976416+0000","last_fullsized":"2024-01-11T10:08:34.976416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'361","reported_seq":373,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:02.107554+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:09:02.107554+0000","last_peered":"2024-01-11T10:09:02.107554+0000","last_clean":"2024-01-11T10:09:02.107554+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:09:02.107554+0000","last_undegraded":"2024-01-11T10:09:02.107554+0000","last_fullsized":"2024-01-11T10:09:02.107554+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23724032,"num_objects":362,"num_object_clones":0,"num_object_copies":1086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":362,"num_write_kb":23168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052675+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052675+0000","last_peered":"2024-01-11T10:08:35.052675+0000","last_clean":"2024-01-11T10:08:35.052675+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052675+0000","last_undegraded":"2024-01-11T10:08:35.052675+0000","last_fullsized":"2024-01-11T10:08:35.052675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'410","reported_seq":422,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:02.712361+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:09:02.712361+0000","last_peered":"2024-01-11T10:09:02.712361+0000","last_clean":"2024-01-11T10:09:02.712361+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:09:02.712361+0000","last_undegraded":"2024-01-11T10:09:02.712361+0000","last_fullsized":"2024-01-11T10:09:02.712361+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26935296,"num_objects":411,"num_object_clones":0,"num_object_copies":1233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":411,"num_write_kb":26304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'391","reported_seq":403,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:01.930308+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:09:01.930308+0000","last_peered":"2024-01-11T10:09:01.930308+0000","last_clean":"2024-01-11T10:09:01.930308+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:09:01.930308+0000","last_undegraded":"2024-01-11T10:09:01.930308+0000","last_fullsized":"2024-01-11T10:09:01.930308+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25690112,"num_objects":392,"num_object_clones":0,"num_object_copies":1176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":392,"num_write_kb":25088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.073743+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:35.073743+0000","last_peered":"2024-01-11T10:08:35.073743+0000","last_clean":"2024-01-11T10:08:35.073743+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:35.073743+0000","last_undegraded":"2024-01-11T10:08:35.073743+0000","last_fullsized":"2024-01-11T10:08:35.073743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'385","reported_seq":397,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.078864+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:09:03.078864+0000","last_peered":"2024-01-11T10:09:03.078864+0000","last_clean":"2024-01-11T10:09:03.078864+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:09:03.078864+0000","last_undegraded":"2024-01-11T10:09:03.078864+0000","last_fullsized":"2024-01-11T10:09:03.078864+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25296896,"num_objects":386,"num_object_clones":0,"num_object_copies":1158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":386,"num_write_kb":24704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'367","reported_seq":379,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:01.879496+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:09:01.879496+0000","last_peered":"2024-01-11T10:09:01.879496+0000","last_clean":"2024-01-11T10:09:01.879496+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:09:01.879496+0000","last_undegraded":"2024-01-11T10:09:01.879496+0000","last_fullsized":"2024-01-11T10:09:01.879496+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24117248,"num_objects":368,"num_object_clones":0,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":368,"num_write_kb":23552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.071023+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:35.071023+0000","last_peered":"2024-01-11T10:08:35.071023+0000","last_clean":"2024-01-11T10:08:35.071023+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:35.071023+0000","last_undegraded":"2024-01-11T10:08:35.071023+0000","last_fullsized":"2024-01-11T10:08:35.071023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'373","reported_seq":385,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:03.096850+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:09:03.096850+0000","last_peered":"2024-01-11T10:09:03.096850+0000","last_clean":"2024-01-11T10:09:03.096850+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:09:03.096850+0000","last_undegraded":"2024-01-11T10:09:03.096850+0000","last_fullsized":"2024-01-11T10:09:03.096850+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24510464,"num_objects":374,"num_object_clones":0,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":374,"num_write_kb":23936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.984172+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:34.984172+0000","last_peered":"2024-01-11T10:08:34.984172+0000","last_clean":"2024-01-11T10:08:34.984172+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:34.984172+0000","last_undegraded":"2024-01-11T10:08:34.984172+0000","last_fullsized":"2024-01-11T10:08:34.984172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":396558336,"num_objects":6051,"num_object_clones":0,"num_object_copies":18153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6051,"num_write_kb":387264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6035,"ondisk_log_size":6035,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43296,"kb_used_data":16472,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94328544,"statfs":{"total":96636764160,"available":96592429056,"internally_reserved":0,"allocated":16867328,"data_stored":267232021,"data_compressed":12688853,"data_compressed_allocated":16703488,"data_compressed_original":267173888,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":43520,"kb_used_data":16696,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94328320,"statfs":{"total":96636764160,"available":96592199680,"internally_reserved":0,"allocated":17096704,"data_stored":270902037,"data_compressed":12863087,"data_compressed_allocated":16932864,"data_compressed_original":270843904,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51228,"kb_used_data":20244,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94320612,"statfs":{"total":96636764160,"available":96584306688,"internally_reserved":0,"allocated":20729856,"data_stored":329032469,"data_compressed":15623207,"data_compressed_allocated":20566016,"data_compressed_original":328974336,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":44140,"kb_used_data":17316,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94327700,"statfs":{"total":96636764160,"available":96591564800,"internally_reserved":0,"allocated":17731584,"data_stored":281452789,"data_compressed":13362088,"data_compressed_allocated":17588224,"data_compressed_original":281411584,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:09:06.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 31 16 6772 6756 13.6199 13.875 0.132472 0.0732991 2024-01-11T10:09:06.609 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:06.730 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:06.744+0000 7fab6746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:06.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:06.751+0000 7fab6746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:07.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 32 16 6977 6961 13.5946 12.8125 0.0729099 0.0734472 2024-01-11T10:09:08.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 33 16 7189 7173 13.5841 13.25 0.0796254 0.0735187 2024-01-11T10:09:09.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:09.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 34 16 7389 7373 13.5522 12.5 0.0794857 0.0736776 2024-01-11T10:09:09.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:09.161+0000 7fbc0399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:09.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:09.167+0000 7fbc0399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:10.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 35 16 7600 7584 13.5417 13.1875 0.0670007 0.0737541 2024-01-11T10:09:10.781 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:10.781 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:10.781 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:09:10.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:09:10.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:10.910+0000 7f195bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:10.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:10.919+0000 7f195bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:10.919 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:10.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:10.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:10.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:10.920 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:10.920 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:10.922 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:10.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:09:11.024 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 36 16 7799 7783 13.511 12.4375 0.173313 0.0739015 2024-01-11T10:09:11.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.048+0000 7fcced904640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.042 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.057+0000 7fcced904640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:11.082 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:11.083 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:11.085 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:11.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:11.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.216+0000 7f2f88c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.225+0000 7f2f88c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.244 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:11.245 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:11.248 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:11.248 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:11.248 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:11.372 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.380+0000 7f5566ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.390+0000 7f5566ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:11.434 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:11.434 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:11.434 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:11.435 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:11.437 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:11.528 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.542+0000 7f612235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:11.533 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:11.547+0000 7f612235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:12.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 37 16 8007 7991 13.4972 13 0.0741044 0.0739675 2024-01-11T10:09:13.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 38 16 8207 8191 13.4709 12.5 0.135203 0.0741358 2024-01-11T10:09:13.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:13.856 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:13.869+0000 7f5fc48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:13.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:13.874+0000 7f5fc48ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:14.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 39 16 8415 8399 13.4588 13 0.0550871 0.0742065 2024-01-11T10:09:15.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:09:15.041441+0000 min lat: 0.0112295 max lat: 0.202517 avg lat: 0.0744133 2024-01-11T10:09:15.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:09:15.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 40 16 8606 8590 13.4208 11.9375 0.0478887 0.0744133 2024-01-11T10:09:16.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 41 16 8810 8794 13.4044 12.75 0.050804 0.0745125 2024-01-11T10:09:16.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:16.209 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.223+0000 7f129a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.228+0000 7f129a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:16.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:16.438 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:09:16.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:09:16.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.587+0000 7f28e4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.582 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.597+0000 7f28e4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:16.644 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:16.647 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:16.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:09:16.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.782+0000 7f1917817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.791+0000 7f1917817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:16.826 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:16.829 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:16.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:16.950 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.959+0000 7f052062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:16.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:16.969+0000 7f052062d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:17.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 42 16 9013 8997 13.3873 12.6875 0.0626056 0.0746207 2024-01-11T10:09:17.038 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:17.038 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:17.038 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:17.038 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:17.038 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:17.038 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:17.041 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:17.041 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:17.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:17.189+0000 7f543539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:17.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:17.198+0000 7f543539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:17.256 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:17.257 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:17.257 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:17.257 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:17.257 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:17.260 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:18.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 43 16 9239 9223 13.4044 14.125 0.0407124 0.0745314 2024-01-11T10:09:18.422 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:18.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:18.577+0000 7f445ec64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:18.568 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:18.583+0000 7f445ec64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:19.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 44 16 9429 9413 13.3697 11.875 0.152616 0.0746978 2024-01-11T10:09:20.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 45 16 9640 9624 13.3656 13.1875 0.0522113 0.0747622 2024-01-11T10:09:20.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:20.829 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:20.842+0000 7f614d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:20.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:20.847+0000 7f614d86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:21.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 46 16 9847 9831 13.3563 12.9375 0.0982385 0.0747761 2024-01-11T10:09:22.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 47 16 10043 10027 13.3327 12.25 0.0827133 0.0749159 2024-01-11T10:09:22.261 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:22.261 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:22.261 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:09:22.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:09:22.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.391+0000 7fb5b531e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.399+0000 7fb5b531e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:22.402 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:22.405 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:22.405 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:09:22.525 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.535+0000 7f4095f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.544+0000 7f4095f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:22.602 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:22.605 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:22.605 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:22.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.738+0000 7f48c3f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.747+0000 7f48c3f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:22.764 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:22.765 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:22.765 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:22.767 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:22.767 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:22.768 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:22.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.911+0000 7f07c7064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:22.920+0000 7f07c7064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:22.974 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:22.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:22.977 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:23.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:23.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 48 16 10247 10231 13.3205 12.75 0.0681855 0.0749895 2024-01-11T10:09:23.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:23.146+0000 7fea1362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:23.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:23.152+0000 7fea1362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:24.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 49 16 10458 10442 13.3178 13.1875 0.0512196 0.0750128 2024-01-11T10:09:25.025 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 50 16 10667 10651 13.3127 13.0625 0.0354715 0.0750521 2024-01-11T10:09:25.371 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:25.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:25.499+0000 7fa4c346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:25.490 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:25.505+0000 7fa4c346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:26.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 51 16 10867 10851 13.2967 12.5 0.0868209 0.0751509 2024-01-11T10:09:27.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 52 16 11083 11067 13.3006 13.5 0.0703695 0.0751205 2024-01-11T10:09:27.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:27.795 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:27.809+0000 7fa1f979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:27.799 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:27.814+0000 7fa1f979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:27.978 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:27.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:27.978 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:09:27.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:09:28.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 53 16 11291 11275 13.2949 13 0.0418285 0.0751525 2024-01-11T10:09:28.103 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.113+0000 7fdb0479e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.107 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.122+0000 7fdb0479e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.146 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:28.146 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:28.146 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:28.146 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:28.146 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:28.147 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:28.149 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:28.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:09:28.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.337+0000 7fb6f6b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.347+0000 7fb6f6b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.385 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:28.385 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:28.385 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:28.385 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:28.386 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:28.388 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:28.389 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:28.519 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.528+0000 7f6466f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.522 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.538+0000 7f6466f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.614 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:28.614 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:28.615 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:28.617 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:28.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:28.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.746+0000 7f5d720de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:28.757+0000 7f5d720de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:28.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:28.804 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:28.804 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:28.804 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:28.806 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:29.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 54 16 11498 11482 13.2883 12.9375 0.0478917 0.0751994 2024-01-11T10:09:30.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 55 16 11698 11682 13.2739 12.5 0.0971753 0.075277 2024-01-11T10:09:30.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:30.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:30.227+0000 7ff416d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:30.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:30.231+0000 7ff416d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:31.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 56 16 11901 11885 13.2634 12.6875 0.0896738 0.0753287 2024-01-11T10:09:32.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 57 16 12108 12092 13.2577 12.9375 0.0583208 0.0753613 2024-01-11T10:09:32.429 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:32.547 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:32.560+0000 7f527dcf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:32.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:32.565+0000 7f527dcf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:33.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 58 16 12290 12274 13.2252 11.375 0.11365 0.0755413 2024-01-11T10:09:33.807 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:33.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:33.807 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:09:33.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:09:33.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:33.939+0000 7f7ed146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:33.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:33.948+0000 7f7ed146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:33.952 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:33.952 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:33.952 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:33.952 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:33.952 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:33.953 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:33.955 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:33.955 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:33.956 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:09:34.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 59 16 12493 12477 13.2161 12.6875 0.0871023 0.0755716 2024-01-11T10:09:34.085 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.096+0000 7ff4abcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.090 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.105+0000 7ff4abcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:34.151 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:34.152 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:34.154 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:34.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:34.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.295+0000 7f7819e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.303+0000 7f7819e20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.311 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:34.311 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:34.311 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:34.311 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:34.311 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:34.312 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:34.315 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:34.315 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:34.433 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.443+0000 7fdd6815e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.437 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.452+0000 7fdd6815e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:34.510 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:34.511 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:34.513 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:34.514 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:34.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:34.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.909+0000 7f8af391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:34.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:34.914+0000 7f8af391e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:35.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:09:35.043146+0000 min lat: 0.0112295 max lat: 0.202517 avg lat: 0.0757216 2024-01-11T10:09:35.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:09:35.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 60 16 12685 12669 13.1958 12 0.0793517 0.0757216 2024-01-11T10:09:36.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 61 16 12889 12873 13.1885 12.75 0.0358544 0.0757432 2024-01-11T10:09:37.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 62 16 13094 13078 13.1824 12.8125 0.140036 0.075795 2024-01-11T10:09:37.083 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:37.215 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:37.229+0000 7f82dee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:37.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:37.235+0000 7f82dee2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:38.026 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 63 16 13281 13265 13.1586 11.6875 0.0359665 0.0759158 2024-01-11T10:09:39.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 64 15 13477 13462 13.1454 12.3125 0.0864504 0.076019 2024-01-11T10:09:39.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:39.514 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:39.514 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:39.515 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:09:39.515 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:09:39.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:39.555+0000 7fdd04d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:39.549 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:39.564+0000 7fdd04d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:39.643 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:39.652+0000 7f0b1c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:39.647 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:39.662+0000 7f0b1c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:39.689 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:39.690 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:39.692 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:39.692 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:39.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:09:39.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:39.869+0000 7f00e02f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:39.864 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:39.879+0000 7f00e02f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:39.906 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:39.907 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:39.909 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:39.909 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:39.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:09:40.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 65 16 13676 13660 13.1335 12.375 0.105138 0.076077 2024-01-11T10:09:40.029 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:40.039+0000 7fd4b639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:40.033 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:40.048+0000 7fd4b639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:40.048 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:40.050 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:40.051 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:40.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:09:40.191 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:40.201+0000 7f758502d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:40.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:40.209+0000 7f758502d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:09:40.238 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:09:40.239 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:09:40.239 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:09:40.241 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:09:40.241 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:09:41.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 66 16 13875 13859 13.123 12.4375 0.0527294 0.07615 2024-01-11T10:09:41.804 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:41.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:41.934+0000 7f0d0faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:41.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:41.940+0000 7f0d0faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:42.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 67 16 14075 14059 13.1137 12.5 0.13151 0.0762043 2024-01-11T10:09:43.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 68 16 14244 14228 13.0761 10.5625 0.0675236 0.0763884 2024-01-11T10:09:44.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 69 16 14440 14424 13.0641 12.25 0.0907318 0.0765036 2024-01-11T10:09:44.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:44.337 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:44.351+0000 7f7b40f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:44.341 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:44.356+0000 7f7b40f04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:45.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 70 16 14645 14629 13.0605 12.8125 0.0843873 0.0765289 2024-01-11T10:09:45.241 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:45.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:45.242 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:09:45.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:45.374+0000 7f599839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:45.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:45.382+0000 7f599839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:45.614 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:09:45.614 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:09:45.626 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-01-11T10:09:44.112515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":930087456,"num_objects":14185,"num_object_clones":0,"num_object_copies":42555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":14320,"num_write_kb":909606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14264,"ondisk_log_size":14264,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":314876,"kb_used_data":168732,"kb_used_omap":0,"kb_used_meta":146112,"kb_avail":377172484,"statfs":{"total":386547056640,"available":386224623616,"internally_reserved":0,"allocated":172781568,"data_stored":2754316852,"data_compressed":130784344,"data_compressed_allocated":172146688,"data_compressed_original":2754101248,"omap_allocated":0,"internal_metadata":149618688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":194904064,"num_objects":2974,"num_object_clones":0,"num_object_copies":8922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2974,"num_write_kb":190336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001944"},"pg_stats":[{"pgid":"3.f","version":"17'876","reported_seq":888,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:43.131973+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:09:43.131973+0000","last_peered":"2024-01-11T10:09:43.131973+0000","last_clean":"2024-01-11T10:09:43.131973+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:09:43.131973+0000","last_undegraded":"2024-01-11T10:09:43.131973+0000","last_fullsized":"2024-01-11T10:09:43.131973+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57475072,"num_objects":877,"num_object_clones":0,"num_object_copies":2631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":877,"num_write_kb":56128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'885","reported_seq":897,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.939205+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:09:42.939205+0000","last_peered":"2024-01-11T10:09:42.939205+0000","last_clean":"2024-01-11T10:09:42.939205+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:09:42.939205+0000","last_undegraded":"2024-01-11T10:09:42.939205+0000","last_fullsized":"2024-01-11T10:09:42.939205+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58064896,"num_objects":886,"num_object_clones":0,"num_object_copies":2658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":886,"num_write_kb":56704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'858","reported_seq":870,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.011237+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:09:42.011237+0000","last_peered":"2024-01-11T10:09:42.011237+0000","last_clean":"2024-01-11T10:09:42.011237+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:09:42.011237+0000","last_undegraded":"2024-01-11T10:09:42.011237+0000","last_fullsized":"2024-01-11T10:09:42.011237+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56295424,"num_objects":859,"num_object_clones":0,"num_object_copies":2577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":859,"num_write_kb":54976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'875","reported_seq":887,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.850823+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:09:42.850823+0000","last_peered":"2024-01-11T10:09:42.850823+0000","last_clean":"2024-01-11T10:09:42.850823+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:09:42.850823+0000","last_undegraded":"2024-01-11T10:09:42.850823+0000","last_fullsized":"2024-01-11T10:09:42.850823+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57409536,"num_objects":876,"num_object_clones":0,"num_object_copies":2628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":876,"num_write_kb":56064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'866","reported_seq":878,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:41.820072+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:09:41.820072+0000","last_peered":"2024-01-11T10:09:41.820072+0000","last_clean":"2024-01-11T10:09:41.820072+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:09:41.820072+0000","last_undegraded":"2024-01-11T10:09:41.820072+0000","last_fullsized":"2024-01-11T10:09:41.820072+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56819712,"num_objects":867,"num_object_clones":0,"num_object_copies":2601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":867,"num_write_kb":55488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'898","reported_seq":910,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.758827+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:09:42.758827+0000","last_peered":"2024-01-11T10:09:42.758827+0000","last_clean":"2024-01-11T10:09:42.758827+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:09:42.758827+0000","last_undegraded":"2024-01-11T10:09:42.758827+0000","last_fullsized":"2024-01-11T10:09:42.758827+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":898,"log_dups_size":0,"ondisk_log_size":898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58916864,"num_objects":899,"num_object_clones":0,"num_object_copies":2697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":899,"num_write_kb":57536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'886","reported_seq":898,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:43.152242+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:09:43.152242+0000","last_peered":"2024-01-11T10:09:43.152242+0000","last_clean":"2024-01-11T10:09:43.152242+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:09:43.152242+0000","last_undegraded":"2024-01-11T10:09:43.152242+0000","last_fullsized":"2024-01-11T10:09:43.152242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58130432,"num_objects":887,"num_object_clones":0,"num_object_copies":2661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":887,"num_write_kb":56768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'872","reported_seq":884,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.108740+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:09:42.108740+0000","last_peered":"2024-01-11T10:09:42.108740+0000","last_clean":"2024-01-11T10:09:42.108740+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:09:42.108740+0000","last_undegraded":"2024-01-11T10:09:42.108740+0000","last_fullsized":"2024-01-11T10:09:42.108740+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":872,"log_dups_size":0,"ondisk_log_size":872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57212928,"num_objects":873,"num_object_clones":0,"num_object_copies":2619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":873,"num_write_kb":55872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.992304+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:34.992304+0000","last_peered":"2024-01-11T10:08:34.992304+0000","last_clean":"2024-01-11T10:08:34.992304+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:34.992304+0000","last_undegraded":"2024-01-11T10:08:34.992304+0000","last_fullsized":"2024-01-11T10:08:34.992304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'895","reported_seq":907,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:41.947534+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:09:41.947534+0000","last_peered":"2024-01-11T10:09:41.947534+0000","last_clean":"2024-01-11T10:09:41.947534+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:09:41.947534+0000","last_undegraded":"2024-01-11T10:09:41.947534+0000","last_fullsized":"2024-01-11T10:09:41.947534+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":896,"num_object_clones":0,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":896,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'927","reported_seq":939,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.094610+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:09:42.094610+0000","last_peered":"2024-01-11T10:09:42.094610+0000","last_clean":"2024-01-11T10:09:42.094610+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:09:42.094610+0000","last_undegraded":"2024-01-11T10:09:42.094610+0000","last_fullsized":"2024-01-11T10:09:42.094610+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60817408,"num_objects":928,"num_object_clones":0,"num_object_copies":2784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":59392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.976416+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:34.976416+0000","last_peered":"2024-01-11T10:08:34.976416+0000","last_clean":"2024-01-11T10:08:34.976416+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:34.976416+0000","last_undegraded":"2024-01-11T10:08:34.976416+0000","last_fullsized":"2024-01-11T10:08:34.976416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'845","reported_seq":857,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.032877+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:09:42.032877+0000","last_peered":"2024-01-11T10:09:42.032877+0000","last_clean":"2024-01-11T10:09:42.032877+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:09:42.032877+0000","last_undegraded":"2024-01-11T10:09:42.032877+0000","last_fullsized":"2024-01-11T10:09:42.032877+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55443456,"num_objects":846,"num_object_clones":0,"num_object_copies":2538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":846,"num_write_kb":54144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052675+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052675+0000","last_peered":"2024-01-11T10:08:35.052675+0000","last_clean":"2024-01-11T10:08:35.052675+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052675+0000","last_undegraded":"2024-01-11T10:08:35.052675+0000","last_fullsized":"2024-01-11T10:08:35.052675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'963","reported_seq":975,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:43.097490+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:09:43.097490+0000","last_peered":"2024-01-11T10:09:43.097490+0000","last_clean":"2024-01-11T10:09:43.097490+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:09:43.097490+0000","last_undegraded":"2024-01-11T10:09:43.097490+0000","last_fullsized":"2024-01-11T10:09:43.097490+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":963,"log_dups_size":0,"ondisk_log_size":963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176704,"num_objects":964,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":964,"num_write_kb":61696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'862","reported_seq":874,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:42.015523+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:09:42.015523+0000","last_peered":"2024-01-11T10:09:42.015523+0000","last_clean":"2024-01-11T10:09:42.015523+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:09:42.015523+0000","last_undegraded":"2024-01-11T10:09:42.015523+0000","last_fullsized":"2024-01-11T10:09:42.015523+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56557568,"num_objects":863,"num_object_clones":0,"num_object_copies":2589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":55232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.073743+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:35.073743+0000","last_peered":"2024-01-11T10:08:35.073743+0000","last_clean":"2024-01-11T10:08:35.073743+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:35.073743+0000","last_undegraded":"2024-01-11T10:08:35.073743+0000","last_fullsized":"2024-01-11T10:08:35.073743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'914","reported_seq":926,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:43.048348+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:09:43.048348+0000","last_peered":"2024-01-11T10:09:43.048348+0000","last_clean":"2024-01-11T10:09:43.048348+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:09:43.048348+0000","last_undegraded":"2024-01-11T10:09:43.048348+0000","last_fullsized":"2024-01-11T10:09:43.048348+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":914,"log_dups_size":0,"ondisk_log_size":914,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59965440,"num_objects":915,"num_object_clones":0,"num_object_copies":2745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":915,"num_write_kb":58560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'879","reported_seq":891,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:41.980197+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:09:41.980197+0000","last_peered":"2024-01-11T10:09:41.980197+0000","last_clean":"2024-01-11T10:09:41.980197+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:09:41.980197+0000","last_undegraded":"2024-01-11T10:09:41.980197+0000","last_fullsized":"2024-01-11T10:09:41.980197+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57671680,"num_objects":880,"num_object_clones":0,"num_object_copies":2640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":880,"num_write_kb":56320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.071023+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:35.071023+0000","last_peered":"2024-01-11T10:08:35.071023+0000","last_clean":"2024-01-11T10:08:35.071023+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:35.071023+0000","last_undegraded":"2024-01-11T10:08:35.071023+0000","last_fullsized":"2024-01-11T10:08:35.071023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'866","reported_seq":878,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:43.134103+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:09:43.134103+0000","last_peered":"2024-01-11T10:09:43.134103+0000","last_clean":"2024-01-11T10:09:43.134103+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:09:43.134103+0000","last_undegraded":"2024-01-11T10:09:43.134103+0000","last_fullsized":"2024-01-11T10:09:43.134103+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56819712,"num_objects":867,"num_object_clones":0,"num_object_copies":2601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":867,"num_write_kb":55488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.984172+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:34.984172+0000","last_peered":"2024-01-11T10:08:34.984172+0000","last_clean":"2024-01-11T10:08:34.984172+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:34.984172+0000","last_undegraded":"2024-01-11T10:08:34.984172+0000","last_fullsized":"2024-01-11T10:08:34.984172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":929497088,"num_objects":14183,"num_object_clones":0,"num_object_copies":42549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14183,"num_write_kb":907712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14167,"ondisk_log_size":14167,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":69864,"kb_used_data":38880,"kb_used_omap":0,"kb_used_meta":30976,"kb_avail":94301976,"statfs":{"total":96636764160,"available":96565223424,"internally_reserved":0,"allocated":39813120,"data_stored":634364693,"data_compressed":30122277,"data_compressed_allocated":39649280,"data_compressed_original":634306560,"omap_allocated":0,"internal_metadata":31719424},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705749,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74484,"kb_used_data":39340,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94297356,"statfs":{"total":96636764160,"available":96560492544,"internally_reserved":0,"allocated":40284160,"data_stored":641901333,"data_compressed":30480118,"data_compressed_allocated":40120320,"data_compressed_original":641843200,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93736,"kb_used_data":48864,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94278104,"statfs":{"total":96636764160,"available":96540778496,"internally_reserved":0,"allocated":50036736,"data_stored":797942549,"data_compressed":37889566,"data_compressed_allocated":49872896,"data_compressed_original":797884416,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705748,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76792,"kb_used_data":41648,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94295048,"statfs":{"total":96636764160,"available":96558129152,"internally_reserved":0,"allocated":42647552,"data_stored":680108277,"data_compressed":32292383,"data_compressed_allocated":42504192,"data_compressed_original":680067072,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:09:46.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 71 16 14845 14829 13.0526 12.5 0.0280195 0.0765528 2024-01-11T10:09:46.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:46.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:46.829+0000 7fc7ae26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:46.820 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:46.835+0000 7fc7ae26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:47.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 72 16 15028 15012 13.0302 11.4375 0.0245715 0.0767053 2024-01-11T10:09:48.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 73 16 15225 15209 13.0203 12.3125 0.0519169 0.0767409 2024-01-11T10:09:49.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 74 16 15427 15411 13.015 12.625 0.0930173 0.0767853 2024-01-11T10:09:49.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:49.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:49.235+0000 7f88b6624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:49.225 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:49.240+0000 7f88b6624640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:50.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 75 16 15622 15606 13.0039 12.1875 0.0971555 0.0768467 2024-01-11T10:09:50.628 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:50.628 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:09:50.629 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:09:50.747 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:50.758+0000 7f62a8d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:50.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:50.765+0000 7f62a8d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:50.959 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:09:50.960 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:09:50.974 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-01-11T10:09:50.113470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":993133088,"num_objects":15147,"num_object_clones":0,"num_object_copies":45441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":15282,"num_write_kb":971174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15226,"ondisk_log_size":15226,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":338904,"kb_used_data":180280,"kb_used_omap":0,"kb_used_meta":158592,"kb_avail":377148456,"statfs":{"total":386547056640,"available":386200018944,"internally_reserved":0,"allocated":184606720,"data_stored":2943519284,"data_compressed":139768688,"data_compressed_allocated":183971840,"data_compressed_original":2943303680,"omap_allocated":0,"internal_metadata":162398208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":160038912,"num_objects":2442,"num_object_clones":0,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2442,"num_write_kb":156288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001996"},"pg_stats":[{"pgid":"3.f","version":"17'933","reported_seq":945,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:48.139754+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:09:48.139754+0000","last_peered":"2024-01-11T10:09:48.139754+0000","last_clean":"2024-01-11T10:09:48.139754+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:09:48.139754+0000","last_undegraded":"2024-01-11T10:09:48.139754+0000","last_fullsized":"2024-01-11T10:09:48.139754+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61210624,"num_objects":934,"num_object_clones":0,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":59776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"17'970","reported_seq":982,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:48.072168+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:09:48.072168+0000","last_peered":"2024-01-11T10:09:48.072168+0000","last_clean":"2024-01-11T10:09:48.072168+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:09:48.072168+0000","last_undegraded":"2024-01-11T10:09:48.072168+0000","last_fullsized":"2024-01-11T10:09:48.072168+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63635456,"num_objects":971,"num_object_clones":0,"num_object_copies":2913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":971,"num_write_kb":62144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'913","reported_seq":925,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:46.939872+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:09:46.939872+0000","last_peered":"2024-01-11T10:09:46.939872+0000","last_clean":"2024-01-11T10:09:46.939872+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:09:46.939872+0000","last_undegraded":"2024-01-11T10:09:46.939872+0000","last_fullsized":"2024-01-11T10:09:46.939872+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":913,"log_dups_size":0,"ondisk_log_size":913,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59899904,"num_objects":914,"num_object_clones":0,"num_object_copies":2742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":914,"num_write_kb":58496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'939","reported_seq":951,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:48.098984+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:09:48.098984+0000","last_peered":"2024-01-11T10:09:48.098984+0000","last_clean":"2024-01-11T10:09:48.098984+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:09:48.098984+0000","last_undegraded":"2024-01-11T10:09:48.098984+0000","last_fullsized":"2024-01-11T10:09:48.098984+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":940,"num_write_kb":60160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'918","reported_seq":930,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.032529+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:09:47.032529+0000","last_peered":"2024-01-11T10:09:47.032529+0000","last_clean":"2024-01-11T10:09:47.032529+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:09:47.032529+0000","last_undegraded":"2024-01-11T10:09:47.032529+0000","last_fullsized":"2024-01-11T10:09:47.032529+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":918,"log_dups_size":0,"ondisk_log_size":918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60227584,"num_objects":919,"num_object_clones":0,"num_object_copies":2757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":919,"num_write_kb":58816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'966","reported_seq":978,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.932299+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:09:47.932299+0000","last_peered":"2024-01-11T10:09:47.932299+0000","last_clean":"2024-01-11T10:09:47.932299+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:09:47.932299+0000","last_undegraded":"2024-01-11T10:09:47.932299+0000","last_fullsized":"2024-01-11T10:09:47.932299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":966,"log_dups_size":0,"ondisk_log_size":966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":967,"num_write_kb":61888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'943","reported_seq":955,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:48.058105+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:09:48.058105+0000","last_peered":"2024-01-11T10:09:48.058105+0000","last_clean":"2024-01-11T10:09:48.058105+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:09:48.058105+0000","last_undegraded":"2024-01-11T10:09:48.058105+0000","last_fullsized":"2024-01-11T10:09:48.058105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":943,"log_dups_size":0,"ondisk_log_size":943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61865984,"num_objects":944,"num_object_clones":0,"num_object_copies":2832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":944,"num_write_kb":60416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"17'925","reported_seq":937,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.045540+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:09:47.045540+0000","last_peered":"2024-01-11T10:09:47.045540+0000","last_clean":"2024-01-11T10:09:47.045540+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:09:47.045540+0000","last_undegraded":"2024-01-11T10:09:47.045540+0000","last_fullsized":"2024-01-11T10:09:47.045540+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60686336,"num_objects":926,"num_object_clones":0,"num_object_copies":2778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":926,"num_write_kb":59264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.992304+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:08:34.992304+0000","last_peered":"2024-01-11T10:08:34.992304+0000","last_clean":"2024-01-11T10:08:34.992304+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:08:34.992304+0000","last_undegraded":"2024-01-11T10:08:34.992304+0000","last_fullsized":"2024-01-11T10:08:34.992304+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"17'953","reported_seq":965,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.066664+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:09:47.066664+0000","last_peered":"2024-01-11T10:09:47.066664+0000","last_clean":"2024-01-11T10:09:47.066664+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:09:47.066664+0000","last_undegraded":"2024-01-11T10:09:47.066664+0000","last_fullsized":"2024-01-11T10:09:47.066664+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":953,"log_dups_size":0,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62521344,"num_objects":954,"num_object_clones":0,"num_object_copies":2862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":954,"num_write_kb":61056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'987","reported_seq":999,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:46.909113+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:09:46.909113+0000","last_peered":"2024-01-11T10:09:46.909113+0000","last_clean":"2024-01-11T10:09:46.909113+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:09:46.909113+0000","last_undegraded":"2024-01-11T10:09:46.909113+0000","last_fullsized":"2024-01-11T10:09:46.909113+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":63232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.976416+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:08:34.976416+0000","last_peered":"2024-01-11T10:08:34.976416+0000","last_clean":"2024-01-11T10:08:34.976416+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:08:34.976416+0000","last_undegraded":"2024-01-11T10:08:34.976416+0000","last_fullsized":"2024-01-11T10:08:34.976416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'899","reported_seq":911,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.026727+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:09:47.026727+0000","last_peered":"2024-01-11T10:09:47.026727+0000","last_clean":"2024-01-11T10:09:47.026727+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:09:47.026727+0000","last_undegraded":"2024-01-11T10:09:47.026727+0000","last_fullsized":"2024-01-11T10:09:47.026727+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":899,"log_dups_size":0,"ondisk_log_size":899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58982400,"num_objects":900,"num_object_clones":0,"num_object_copies":2700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":900,"num_write_kb":57600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052675+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052675+0000","last_peered":"2024-01-11T10:08:35.052675+0000","last_clean":"2024-01-11T10:08:35.052675+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052675+0000","last_undegraded":"2024-01-11T10:08:35.052675+0000","last_fullsized":"2024-01-11T10:08:35.052675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.983445+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:09:47.983445+0000","last_peered":"2024-01-11T10:09:47.983445+0000","last_clean":"2024-01-11T10:09:47.983445+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:09:47.983445+0000","last_undegraded":"2024-01-11T10:09:47.983445+0000","last_fullsized":"2024-01-11T10:09:47.983445+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"17'917","reported_seq":929,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.085114+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:09:47.085114+0000","last_peered":"2024-01-11T10:09:47.085114+0000","last_clean":"2024-01-11T10:09:47.085114+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:09:47.085114+0000","last_undegraded":"2024-01-11T10:09:47.085114+0000","last_fullsized":"2024-01-11T10:09:47.085114+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60162048,"num_objects":918,"num_object_clones":0,"num_object_copies":2754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918,"num_write_kb":58752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.073743+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:08:35.073743+0000","last_peered":"2024-01-11T10:08:35.073743+0000","last_clean":"2024-01-11T10:08:35.073743+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:08:35.073743+0000","last_undegraded":"2024-01-11T10:08:35.073743+0000","last_fullsized":"2024-01-11T10:08:35.073743+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'968","reported_seq":980,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:48.131405+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:09:48.131405+0000","last_peered":"2024-01-11T10:09:48.131405+0000","last_clean":"2024-01-11T10:09:48.131405+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:09:48.131405+0000","last_undegraded":"2024-01-11T10:09:48.131405+0000","last_fullsized":"2024-01-11T10:09:48.131405+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":968,"log_dups_size":0,"ondisk_log_size":968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63504384,"num_objects":969,"num_object_clones":0,"num_object_copies":2907,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":969,"num_write_kb":62016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'956","reported_seq":968,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.093105+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:09:47.093105+0000","last_peered":"2024-01-11T10:09:47.093105+0000","last_clean":"2024-01-11T10:09:47.093105+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:09:47.093105+0000","last_undegraded":"2024-01-11T10:09:47.093105+0000","last_fullsized":"2024-01-11T10:09:47.093105+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62717952,"num_objects":957,"num_object_clones":0,"num_object_copies":2871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":957,"num_write_kb":61248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.071023+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:08:35.071023+0000","last_peered":"2024-01-11T10:08:35.071023+0000","last_clean":"2024-01-11T10:08:35.071023+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:08:35.071023+0000","last_undegraded":"2024-01-11T10:08:35.071023+0000","last_fullsized":"2024-01-11T10:08:35.071023+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'923","reported_seq":935,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:47.983463+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:09:47.983463+0000","last_peered":"2024-01-11T10:09:47.983463+0000","last_clean":"2024-01-11T10:09:47.983463+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:09:47.983463+0000","last_undegraded":"2024-01-11T10:09:47.983463+0000","last_fullsized":"2024-01-11T10:09:47.983463+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":923,"log_dups_size":0,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60555264,"num_objects":924,"num_object_clones":0,"num_object_copies":2772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":924,"num_write_kb":59136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:34.984172+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:08:34.984172+0000","last_peered":"2024-01-11T10:08:34.984172+0000","last_clean":"2024-01-11T10:08:34.984172+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:08:34.984172+0000","last_undegraded":"2024-01-11T10:08:34.984172+0000","last_fullsized":"2024-01-11T10:08:34.984172+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":992542720,"num_objects":15145,"num_object_clones":0,"num_object_copies":45435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15145,"num_write_kb":969280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15129,"ondisk_log_size":15129,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76664,"kb_used_data":41520,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94295176,"statfs":{"total":96636764160,"available":96558260224,"internally_reserved":0,"allocated":42516480,"data_stored":677618453,"data_compressed":32176197,"data_compressed_allocated":42352640,"data_compressed_original":677560320,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705754,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77092,"kb_used_data":41948,"kb_used_omap":0,"kb_used_meta":35136,"kb_avail":94294748,"statfs":{"total":96636764160,"available":96557821952,"internally_reserved":0,"allocated":42954752,"data_stored":684630805,"data_compressed":32509142,"data_compressed_allocated":42790912,"data_compressed_original":684572672,"omap_allocated":0,"internal_metadata":35979264},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":101280,"kb_used_data":52248,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94270560,"statfs":{"total":96636764160,"available":96533053440,"internally_reserved":0,"allocated":53501952,"data_stored":853386005,"data_compressed":40522318,"data_compressed_allocated":53338112,"data_compressed_original":853327872,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":83868,"kb_used_data":44564,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94287972,"statfs":{"total":96636764160,"available":96550883328,"internally_reserved":0,"allocated":45633536,"data_stored":727884021,"data_compressed":34561031,"data_compressed_allocated":45490176,"data_compressed_original":727842816,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:09:51.027 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 76 16 15816 15800 12.9924 12.125 0.0398048 0.076924 2024-01-11T10:09:51.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:51.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:51.550+0000 7fdfa995e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:51.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:51.555+0000 7fdfa995e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:52.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 77 16 16014 15998 12.9843 12.375 0.029525 0.07695 2024-01-11T10:09:53.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 78 16 16201 16185 12.9677 11.6875 0.109467 0.0770536 2024-01-11T10:09:53.817 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:53.938 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:53.953+0000 7f659802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:53.942 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:53.958+0000 7f659802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:54.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 79 16 16390 16374 12.953 11.8125 0.10147 0.0771525 2024-01-11T10:09:55.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:09:55.044769+0000 min lat: 0.0112295 max lat: 0.217041 avg lat: 0.0772251 2024-01-11T10:09:55.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:09:55.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 80 16 16582 16566 12.9411 12 0.0605846 0.0772251 2024-01-11T10:09:55.976 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:09:55.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:09:56.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 81 16 16772 16756 12.9279 11.875 0.0746413 0.0772962 2024-01-11T10:09:56.094 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:56.103+0000 7febad11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:56.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:56.114+0000 7febad11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:56.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:09:56.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:56.394+0000 7fc13b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:56.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:56.400+0000 7fc13b2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:56.961 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:09:56.976 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:09:57.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 82 16 16969 16953 12.9204 12.3125 0.14127 0.0773539 2024-01-11T10:09:57.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:57.110+0000 7f63bb91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:57.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:57.122+0000 7f63bb91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:57.962 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:09:57.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:09:58.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 83 16 17167 17151 12.9138 12.375 0.101692 0.0773903 2024-01-11T10:09:58.095 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:58.106+0000 7f61e266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:58.100 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:58.116+0000 7f61e266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:58.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:09:58.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:58.773+0000 7f8a8135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:58.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:58.778+0000 7f8a8135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:58.968 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:09:58.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:09:59.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 84 16 17343 17327 12.891 11 0.080264 0.0775135 2024-01-11T10:09:59.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:59.113+0000 7f91722ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:59.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:09:59.122+0000 7f91722ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:09:59.974 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:09:59.989 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:09:59.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:10:00.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 85 16 17534 17518 12.8798 11.9375 0.0839759 0.077587 2024-01-11T10:10:00.148 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:00.161+0000 7f17b98f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:00.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:00.169+0000 7f17b98f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:00.357 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:10:00.358 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:10:00.371 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-01-11T10:10:00.114888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1120338464,"num_objects":17088,"num_object_clones":0,"num_object_copies":51264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":17223,"num_write_kb":1095398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17169,"ondisk_log_size":17169,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":380396,"kb_used_data":203724,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":377106964,"statfs":{"total":386547056640,"available":386157531136,"internally_reserved":0,"allocated":208613376,"data_stored":3327625780,"data_compressed":158008120,"data_compressed_allocated":207978496,"data_compressed_original":3327410176,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":967,"num_write_kb":61888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000513"},"pg_stats":[{"pgid":"3.f","version":"17'1046","reported_seq":1058,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.918297+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:09:57.918297+0000","last_peered":"2024-01-11T10:09:57.918297+0000","last_clean":"2024-01-11T10:09:57.918297+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:09:57.918297+0000","last_undegraded":"2024-01-11T10:09:57.918297+0000","last_fullsized":"2024-01-11T10:09:57.918297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68616192,"num_objects":1047,"num_object_clones":0,"num_object_copies":3141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":67008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'1081","reported_seq":1095,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.103891+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:09:58.103891+0000","last_peered":"2024-01-11T10:09:58.103891+0000","last_clean":"2024-01-11T10:09:58.103891+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:09:58.103891+0000","last_undegraded":"2024-01-11T10:09:58.103891+0000","last_fullsized":"2024-01-11T10:09:58.103891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":69248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'1033","reported_seq":1047,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.021933+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:09:57.021933+0000","last_peered":"2024-01-11T10:09:57.021933+0000","last_clean":"2024-01-11T10:09:57.021933+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:09:57.021933+0000","last_undegraded":"2024-01-11T10:09:57.021933+0000","last_fullsized":"2024-01-11T10:09:57.021933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67764224,"num_objects":1034,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":66176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'1070","reported_seq":1082,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.065276+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:09:58.065276+0000","last_peered":"2024-01-11T10:09:58.065276+0000","last_clean":"2024-01-11T10:09:58.065276+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:09:58.065276+0000","last_undegraded":"2024-01-11T10:09:58.065276+0000","last_fullsized":"2024-01-11T10:09:58.065276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70189056,"num_objects":1071,"num_object_clones":0,"num_object_copies":3213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1071,"num_write_kb":68544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.042379+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:09:57.042379+0000","last_peered":"2024-01-11T10:09:57.042379+0000","last_clean":"2024-01-11T10:09:57.042379+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:09:57.042379+0000","last_undegraded":"2024-01-11T10:09:57.042379+0000","last_fullsized":"2024-01-11T10:09:57.042379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'1085","reported_seq":1097,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.025854+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:09:58.025854+0000","last_peered":"2024-01-11T10:09:58.025854+0000","last_clean":"2024-01-11T10:09:58.025854+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:09:58.025854+0000","last_undegraded":"2024-01-11T10:09:58.025854+0000","last_fullsized":"2024-01-11T10:09:58.025854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":69440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'1069","reported_seq":1083,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.167886+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:09:58.167886+0000","last_peered":"2024-01-11T10:09:58.167886+0000","last_clean":"2024-01-11T10:09:58.167886+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:09:58.167886+0000","last_undegraded":"2024-01-11T10:09:58.167886+0000","last_fullsized":"2024-01-11T10:09:58.167886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70123520,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1070,"num_write_kb":68480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'1056","reported_seq":1070,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.023426+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:09:57.023426+0000","last_peered":"2024-01-11T10:09:57.023426+0000","last_clean":"2024-01-11T10:09:57.023426+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:09:57.023426+0000","last_undegraded":"2024-01-11T10:09:57.023426+0000","last_fullsized":"2024-01-11T10:09:57.023426+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":67648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.024651+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:09:57.024651+0000","last_peered":"2024-01-11T10:09:57.024651+0000","last_clean":"2024-01-11T10:09:57.024651+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:09:57.024651+0000","last_undegraded":"2024-01-11T10:09:57.024651+0000","last_fullsized":"2024-01-11T10:09:57.024651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"18'1099","reported_seq":1113,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.086266+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:09:57.086266+0000","last_peered":"2024-01-11T10:09:57.086266+0000","last_clean":"2024-01-11T10:09:57.086266+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:09:57.086266+0000","last_undegraded":"2024-01-11T10:09:57.086266+0000","last_fullsized":"2024-01-11T10:09:57.086266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1100,"num_write_kb":70400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'1106","reported_seq":1118,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.076280+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:09:57.076280+0000","last_peered":"2024-01-11T10:09:57.076280+0000","last_clean":"2024-01-11T10:09:57.076280+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:09:57.076280+0000","last_undegraded":"2024-01-11T10:09:57.076280+0000","last_fullsized":"2024-01-11T10:09:57.076280+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":3321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1107,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.021385+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:09:57.021385+0000","last_peered":"2024-01-11T10:09:57.021385+0000","last_clean":"2024-01-11T10:09:57.021385+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:09:57.021385+0000","last_undegraded":"2024-01-11T10:09:57.021385+0000","last_fullsized":"2024-01-11T10:09:57.021385+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:56.666700+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:09:56.666700+0000","last_peered":"2024-01-11T10:09:56.666700+0000","last_clean":"2024-01-11T10:09:56.666700+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:09:56.666700+0000","last_undegraded":"2024-01-11T10:09:56.666700+0000","last_fullsized":"2024-01-11T10:09:56.666700+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052837+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052837+0000","last_peered":"2024-01-11T10:08:35.052837+0000","last_clean":"2024-01-11T10:08:35.052837+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052837+0000","last_undegraded":"2024-01-11T10:08:35.052837+0000","last_fullsized":"2024-01-11T10:08:35.052837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'1152","reported_seq":1165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.031816+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:09:58.031816+0000","last_peered":"2024-01-11T10:09:58.031816+0000","last_clean":"2024-01-11T10:09:58.031816+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:09:58.031816+0000","last_undegraded":"2024-01-11T10:09:58.031816+0000","last_fullsized":"2024-01-11T10:09:58.031816+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":1152,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"18'1030","reported_seq":1044,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.036800+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:09:57.036800+0000","last_peered":"2024-01-11T10:09:57.036800+0000","last_clean":"2024-01-11T10:09:57.036800+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:09:57.036800+0000","last_undegraded":"2024-01-11T10:09:57.036800+0000","last_fullsized":"2024-01-11T10:09:57.036800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.053590+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:09:58.053590+0000","last_peered":"2024-01-11T10:09:58.053590+0000","last_clean":"2024-01-11T10:09:58.053590+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:09:58.053590+0000","last_undegraded":"2024-01-11T10:09:58.053590+0000","last_fullsized":"2024-01-11T10:09:58.053590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'1097","reported_seq":1111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.049945+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:09:58.049945+0000","last_peered":"2024-01-11T10:09:58.049945+0000","last_clean":"2024-01-11T10:09:58.049945+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:09:58.049945+0000","last_undegraded":"2024-01-11T10:09:58.049945+0000","last_fullsized":"2024-01-11T10:09:58.049945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1098,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'1073","reported_seq":1085,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.056316+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:09:57.056316+0000","last_peered":"2024-01-11T10:09:57.056316+0000","last_clean":"2024-01-11T10:09:57.056316+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:09:57.056316+0000","last_undegraded":"2024-01-11T10:09:57.056316+0000","last_fullsized":"2024-01-11T10:09:57.056316+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1074,"num_write_kb":68736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.050593+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:09:58.050593+0000","last_peered":"2024-01-11T10:09:58.050593+0000","last_clean":"2024-01-11T10:09:58.050593+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:09:58.050593+0000","last_undegraded":"2024-01-11T10:09:58.050593+0000","last_fullsized":"2024-01-11T10:09:58.050593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'1037","reported_seq":1051,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.019884+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:09:58.019884+0000","last_peered":"2024-01-11T10:09:58.019884+0000","last_clean":"2024-01-11T10:09:58.019884+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:09:58.019884+0000","last_undegraded":"2024-01-11T10:09:58.019884+0000","last_fullsized":"2024-01-11T10:09:58.019884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68026368,"num_objects":1038,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":66432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.022115+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:09:57.022115+0000","last_peered":"2024-01-11T10:09:57.022115+0000","last_clean":"2024-01-11T10:09:57.022115+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:09:57.022115+0000","last_undegraded":"2024-01-11T10:09:57.022115+0000","last_fullsized":"2024-01-11T10:09:57.022115+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1119748096,"num_objects":17086,"num_object_clones":0,"num_object_copies":51258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":1093504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17072,"ondisk_log_size":17072,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86128,"kb_used_data":46824,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285712,"statfs":{"total":96636764160,"available":96548569088,"internally_reserved":0,"allocated":47947776,"data_stored":764519189,"data_compressed":36302709,"data_compressed_allocated":47783936,"data_compressed_original":764461056,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86700,"kb_used_data":47396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285140,"statfs":{"total":96636764160,"available":96547983360,"internally_reserved":0,"allocated":48533504,"data_stored":773890837,"data_compressed":36747686,"data_compressed_allocated":48369664,"data_compressed_original":773832704,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112212,"kb_used_data":59020,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94259628,"statfs":{"total":96636764160,"available":96521859072,"internally_reserved":0,"allocated":60436480,"data_stored":964338453,"data_compressed":45790934,"data_compressed_allocated":60272640,"data_compressed_original":964280320,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95356,"kb_used_data":50484,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94276484,"statfs":{"total":96636764160,"available":96539119616,"internally_reserved":0,"allocated":51695616,"data_stored":824877301,"data_compressed":39166791,"data_compressed_allocated":51552256,"data_compressed_original":824836096,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:10:00.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:10:00.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:00.521+0000 7fa47c75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:00.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:00.528+0000 7fa47c75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:00.811 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:10:00.811 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:10:00.825 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-01-11T10:10:00.114888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1120338464,"num_objects":17088,"num_object_clones":0,"num_object_copies":51264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":17223,"num_write_kb":1095398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17169,"ondisk_log_size":17169,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":380396,"kb_used_data":203724,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":377106964,"statfs":{"total":386547056640,"available":386157531136,"internally_reserved":0,"allocated":208613376,"data_stored":3327625780,"data_compressed":158008120,"data_compressed_allocated":207978496,"data_compressed_original":3327410176,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":967,"num_write_kb":61888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000513"},"pg_stats":[{"pgid":"3.f","version":"17'1046","reported_seq":1058,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.918297+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:09:57.918297+0000","last_peered":"2024-01-11T10:09:57.918297+0000","last_clean":"2024-01-11T10:09:57.918297+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:09:57.918297+0000","last_undegraded":"2024-01-11T10:09:57.918297+0000","last_fullsized":"2024-01-11T10:09:57.918297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68616192,"num_objects":1047,"num_object_clones":0,"num_object_copies":3141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":67008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'1081","reported_seq":1095,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.103891+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:09:58.103891+0000","last_peered":"2024-01-11T10:09:58.103891+0000","last_clean":"2024-01-11T10:09:58.103891+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:09:58.103891+0000","last_undegraded":"2024-01-11T10:09:58.103891+0000","last_fullsized":"2024-01-11T10:09:58.103891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":69248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'1033","reported_seq":1047,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.021933+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:09:57.021933+0000","last_peered":"2024-01-11T10:09:57.021933+0000","last_clean":"2024-01-11T10:09:57.021933+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:09:57.021933+0000","last_undegraded":"2024-01-11T10:09:57.021933+0000","last_fullsized":"2024-01-11T10:09:57.021933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67764224,"num_objects":1034,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":66176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'1070","reported_seq":1082,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.065276+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:09:58.065276+0000","last_peered":"2024-01-11T10:09:58.065276+0000","last_clean":"2024-01-11T10:09:58.065276+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:09:58.065276+0000","last_undegraded":"2024-01-11T10:09:58.065276+0000","last_fullsized":"2024-01-11T10:09:58.065276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70189056,"num_objects":1071,"num_object_clones":0,"num_object_copies":3213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1071,"num_write_kb":68544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.042379+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:09:57.042379+0000","last_peered":"2024-01-11T10:09:57.042379+0000","last_clean":"2024-01-11T10:09:57.042379+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:09:57.042379+0000","last_undegraded":"2024-01-11T10:09:57.042379+0000","last_fullsized":"2024-01-11T10:09:57.042379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'1085","reported_seq":1097,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.025854+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:09:58.025854+0000","last_peered":"2024-01-11T10:09:58.025854+0000","last_clean":"2024-01-11T10:09:58.025854+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:09:58.025854+0000","last_undegraded":"2024-01-11T10:09:58.025854+0000","last_fullsized":"2024-01-11T10:09:58.025854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":69440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'1069","reported_seq":1083,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.167886+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:09:58.167886+0000","last_peered":"2024-01-11T10:09:58.167886+0000","last_clean":"2024-01-11T10:09:58.167886+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:09:58.167886+0000","last_undegraded":"2024-01-11T10:09:58.167886+0000","last_fullsized":"2024-01-11T10:09:58.167886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70123520,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1070,"num_write_kb":68480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'1056","reported_seq":1070,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.023426+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:09:57.023426+0000","last_peered":"2024-01-11T10:09:57.023426+0000","last_clean":"2024-01-11T10:09:57.023426+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:09:57.023426+0000","last_undegraded":"2024-01-11T10:09:57.023426+0000","last_fullsized":"2024-01-11T10:09:57.023426+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":67648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.024651+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:09:57.024651+0000","last_peered":"2024-01-11T10:09:57.024651+0000","last_clean":"2024-01-11T10:09:57.024651+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:09:57.024651+0000","last_undegraded":"2024-01-11T10:09:57.024651+0000","last_fullsized":"2024-01-11T10:09:57.024651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"18'1099","reported_seq":1113,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.086266+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:09:57.086266+0000","last_peered":"2024-01-11T10:09:57.086266+0000","last_clean":"2024-01-11T10:09:57.086266+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:09:57.086266+0000","last_undegraded":"2024-01-11T10:09:57.086266+0000","last_fullsized":"2024-01-11T10:09:57.086266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1100,"num_write_kb":70400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'1106","reported_seq":1118,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.076280+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:09:57.076280+0000","last_peered":"2024-01-11T10:09:57.076280+0000","last_clean":"2024-01-11T10:09:57.076280+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:09:57.076280+0000","last_undegraded":"2024-01-11T10:09:57.076280+0000","last_fullsized":"2024-01-11T10:09:57.076280+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":3321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1107,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.021385+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:09:57.021385+0000","last_peered":"2024-01-11T10:09:57.021385+0000","last_clean":"2024-01-11T10:09:57.021385+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:09:57.021385+0000","last_undegraded":"2024-01-11T10:09:57.021385+0000","last_fullsized":"2024-01-11T10:09:57.021385+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:56.666700+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:09:56.666700+0000","last_peered":"2024-01-11T10:09:56.666700+0000","last_clean":"2024-01-11T10:09:56.666700+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:09:56.666700+0000","last_undegraded":"2024-01-11T10:09:56.666700+0000","last_fullsized":"2024-01-11T10:09:56.666700+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052837+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052837+0000","last_peered":"2024-01-11T10:08:35.052837+0000","last_clean":"2024-01-11T10:08:35.052837+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052837+0000","last_undegraded":"2024-01-11T10:08:35.052837+0000","last_fullsized":"2024-01-11T10:08:35.052837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'1152","reported_seq":1165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.031816+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:09:58.031816+0000","last_peered":"2024-01-11T10:09:58.031816+0000","last_clean":"2024-01-11T10:09:58.031816+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:09:58.031816+0000","last_undegraded":"2024-01-11T10:09:58.031816+0000","last_fullsized":"2024-01-11T10:09:58.031816+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":1152,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"18'1030","reported_seq":1044,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.036800+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:09:57.036800+0000","last_peered":"2024-01-11T10:09:57.036800+0000","last_clean":"2024-01-11T10:09:57.036800+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:09:57.036800+0000","last_undegraded":"2024-01-11T10:09:57.036800+0000","last_fullsized":"2024-01-11T10:09:57.036800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.053590+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:09:58.053590+0000","last_peered":"2024-01-11T10:09:58.053590+0000","last_clean":"2024-01-11T10:09:58.053590+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:09:58.053590+0000","last_undegraded":"2024-01-11T10:09:58.053590+0000","last_fullsized":"2024-01-11T10:09:58.053590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'1097","reported_seq":1111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.049945+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:09:58.049945+0000","last_peered":"2024-01-11T10:09:58.049945+0000","last_clean":"2024-01-11T10:09:58.049945+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:09:58.049945+0000","last_undegraded":"2024-01-11T10:09:58.049945+0000","last_fullsized":"2024-01-11T10:09:58.049945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1098,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'1073","reported_seq":1085,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.056316+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:09:57.056316+0000","last_peered":"2024-01-11T10:09:57.056316+0000","last_clean":"2024-01-11T10:09:57.056316+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:09:57.056316+0000","last_undegraded":"2024-01-11T10:09:57.056316+0000","last_fullsized":"2024-01-11T10:09:57.056316+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1074,"num_write_kb":68736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.050593+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:09:58.050593+0000","last_peered":"2024-01-11T10:09:58.050593+0000","last_clean":"2024-01-11T10:09:58.050593+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:09:58.050593+0000","last_undegraded":"2024-01-11T10:09:58.050593+0000","last_fullsized":"2024-01-11T10:09:58.050593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'1037","reported_seq":1051,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.019884+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:09:58.019884+0000","last_peered":"2024-01-11T10:09:58.019884+0000","last_clean":"2024-01-11T10:09:58.019884+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:09:58.019884+0000","last_undegraded":"2024-01-11T10:09:58.019884+0000","last_fullsized":"2024-01-11T10:09:58.019884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68026368,"num_objects":1038,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":66432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.022115+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:09:57.022115+0000","last_peered":"2024-01-11T10:09:57.022115+0000","last_clean":"2024-01-11T10:09:57.022115+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:09:57.022115+0000","last_undegraded":"2024-01-11T10:09:57.022115+0000","last_fullsized":"2024-01-11T10:09:57.022115+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1119748096,"num_objects":17086,"num_object_clones":0,"num_object_copies":51258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":1093504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17072,"ondisk_log_size":17072,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86128,"kb_used_data":46824,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285712,"statfs":{"total":96636764160,"available":96548569088,"internally_reserved":0,"allocated":47947776,"data_stored":764519189,"data_compressed":36302709,"data_compressed_allocated":47783936,"data_compressed_original":764461056,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86700,"kb_used_data":47396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285140,"statfs":{"total":96636764160,"available":96547983360,"internally_reserved":0,"allocated":48533504,"data_stored":773890837,"data_compressed":36747686,"data_compressed_allocated":48369664,"data_compressed_original":773832704,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112212,"kb_used_data":59020,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94259628,"statfs":{"total":96636764160,"available":96521859072,"internally_reserved":0,"allocated":60436480,"data_stored":964338453,"data_compressed":45790934,"data_compressed_allocated":60272640,"data_compressed_original":964280320,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95356,"kb_used_data":50484,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94276484,"statfs":{"total":96636764160,"available":96539119616,"internally_reserved":0,"allocated":51695616,"data_stored":824877301,"data_compressed":39166791,"data_compressed_allocated":51552256,"data_compressed_original":824836096,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:10:00.827 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:10:00.827 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:00.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:10:00.948 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:00.960+0000 7fcf6d66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:00.952 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:00.968+0000 7fcf6d66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:01.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:01.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 86 16 17715 17699 12.8616 11.3125 0.0701028 0.0776943 2024-01-11T10:10:01.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:01.144+0000 7f658382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:01.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:01.150+0000 7f658382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:01.231 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:10:01.231 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:10:01.244 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-01-11T10:10:00.114888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1120338464,"num_objects":17088,"num_object_clones":0,"num_object_copies":51264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":17223,"num_write_kb":1095398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17169,"ondisk_log_size":17169,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":380396,"kb_used_data":203724,"kb_used_omap":0,"kb_used_meta":176640,"kb_avail":377106964,"statfs":{"total":386547056640,"available":386157531136,"internally_reserved":0,"allocated":208613376,"data_stored":3327625780,"data_compressed":158008120,"data_compressed_allocated":207978496,"data_compressed_original":3327410176,"omap_allocated":0,"internal_metadata":180879360},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":63373312,"num_objects":967,"num_object_clones":0,"num_object_copies":2901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":967,"num_write_kb":61888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000513"},"pg_stats":[{"pgid":"3.f","version":"17'1046","reported_seq":1058,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.918297+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:09:57.918297+0000","last_peered":"2024-01-11T10:09:57.918297+0000","last_clean":"2024-01-11T10:09:57.918297+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:09:57.918297+0000","last_undegraded":"2024-01-11T10:09:57.918297+0000","last_fullsized":"2024-01-11T10:09:57.918297+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1046,"log_dups_size":0,"ondisk_log_size":1046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68616192,"num_objects":1047,"num_object_clones":0,"num_object_copies":3141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1047,"num_write_kb":67008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"19'1081","reported_seq":1095,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.103891+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:09:58.103891+0000","last_peered":"2024-01-11T10:09:58.103891+0000","last_clean":"2024-01-11T10:09:58.103891+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:09:58.103891+0000","last_undegraded":"2024-01-11T10:09:58.103891+0000","last_fullsized":"2024-01-11T10:09:58.103891+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1082,"num_write_kb":69248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"17'1033","reported_seq":1047,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.021933+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:09:57.021933+0000","last_peered":"2024-01-11T10:09:57.021933+0000","last_clean":"2024-01-11T10:09:57.021933+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:09:57.021933+0000","last_undegraded":"2024-01-11T10:09:57.021933+0000","last_fullsized":"2024-01-11T10:09:57.021933+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1033,"log_dups_size":0,"ondisk_log_size":1033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67764224,"num_objects":1034,"num_object_clones":0,"num_object_copies":3102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1034,"num_write_kb":66176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"17'1070","reported_seq":1082,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.065276+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:09:58.065276+0000","last_peered":"2024-01-11T10:09:58.065276+0000","last_clean":"2024-01-11T10:09:58.065276+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:09:58.065276+0000","last_undegraded":"2024-01-11T10:09:58.065276+0000","last_fullsized":"2024-01-11T10:09:58.065276+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1070,"log_dups_size":0,"ondisk_log_size":1070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70189056,"num_objects":1071,"num_object_clones":0,"num_object_copies":3213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1071,"num_write_kb":68544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.042379+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:09:57.042379+0000","last_peered":"2024-01-11T10:09:57.042379+0000","last_clean":"2024-01-11T10:09:57.042379+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:09:57.042379+0000","last_undegraded":"2024-01-11T10:09:57.042379+0000","last_fullsized":"2024-01-11T10:09:57.042379+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"17'1085","reported_seq":1097,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.025854+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:09:58.025854+0000","last_peered":"2024-01-11T10:09:58.025854+0000","last_clean":"2024-01-11T10:09:58.025854+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:09:58.025854+0000","last_undegraded":"2024-01-11T10:09:58.025854+0000","last_fullsized":"2024-01-11T10:09:58.025854+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1085,"log_dups_size":0,"ondisk_log_size":1085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71106560,"num_objects":1085,"num_object_clones":0,"num_object_copies":3255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1085,"num_write_kb":69440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"17'1069","reported_seq":1083,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.167886+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:09:58.167886+0000","last_peered":"2024-01-11T10:09:58.167886+0000","last_clean":"2024-01-11T10:09:58.167886+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:09:58.167886+0000","last_undegraded":"2024-01-11T10:09:58.167886+0000","last_fullsized":"2024-01-11T10:09:58.167886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1069,"log_dups_size":0,"ondisk_log_size":1069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70123520,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1070,"num_write_kb":68480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"18'1056","reported_seq":1070,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.023426+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:09:57.023426+0000","last_peered":"2024-01-11T10:09:57.023426+0000","last_clean":"2024-01-11T10:09:57.023426+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:09:57.023426+0000","last_undegraded":"2024-01-11T10:09:57.023426+0000","last_fullsized":"2024-01-11T10:09:57.023426+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":67648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":148,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.089865+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:08:35.089865+0000","last_peered":"2024-01-11T10:08:35.089865+0000","last_clean":"2024-01-11T10:08:35.089865+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:08:35.089865+0000","last_undegraded":"2024-01-11T10:08:35.089865+0000","last_fullsized":"2024-01-11T10:08:35.089865+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.024651+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:09:57.024651+0000","last_peered":"2024-01-11T10:09:57.024651+0000","last_clean":"2024-01-11T10:09:57.024651+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:09:57.024651+0000","last_undegraded":"2024-01-11T10:09:57.024651+0000","last_fullsized":"2024-01-11T10:09:57.024651+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"18'1099","reported_seq":1113,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.086266+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:09:57.086266+0000","last_peered":"2024-01-11T10:09:57.086266+0000","last_clean":"2024-01-11T10:09:57.086266+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:09:57.086266+0000","last_undegraded":"2024-01-11T10:09:57.086266+0000","last_fullsized":"2024-01-11T10:09:57.086266+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72089600,"num_objects":1100,"num_object_clones":0,"num_object_copies":3300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1100,"num_write_kb":70400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.051199+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:08:35.051199+0000","last_peered":"2024-01-11T10:08:35.051199+0000","last_clean":"2024-01-11T10:08:35.051199+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:08:35.051199+0000","last_undegraded":"2024-01-11T10:08:35.051199+0000","last_fullsized":"2024-01-11T10:08:35.051199+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"17'1106","reported_seq":1118,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.076280+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:09:57.076280+0000","last_peered":"2024-01-11T10:09:57.076280+0000","last_clean":"2024-01-11T10:09:57.076280+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:09:57.076280+0000","last_undegraded":"2024-01-11T10:09:57.076280+0000","last_fullsized":"2024-01-11T10:09:57.076280+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1106,"log_dups_size":0,"ondisk_log_size":1106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72548352,"num_objects":1107,"num_object_clones":0,"num_object_copies":3321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1107,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.021385+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:09:57.021385+0000","last_peered":"2024-01-11T10:09:57.021385+0000","last_clean":"2024-01-11T10:09:57.021385+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:09:57.021385+0000","last_undegraded":"2024-01-11T10:09:57.021385+0000","last_fullsized":"2024-01-11T10:09:57.021385+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"17'1019","reported_seq":1031,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:56.666700+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:09:56.666700+0000","last_peered":"2024-01-11T10:09:56.666700+0000","last_clean":"2024-01-11T10:09:56.666700+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:09:56.666700+0000","last_undegraded":"2024-01-11T10:09:56.666700+0000","last_fullsized":"2024-01-11T10:09:56.666700+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1019,"log_dups_size":0,"ondisk_log_size":1019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66846720,"num_objects":1020,"num_object_clones":0,"num_object_copies":3060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1020,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.052837+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:08:35.052837+0000","last_peered":"2024-01-11T10:08:35.052837+0000","last_clean":"2024-01-11T10:08:35.052837+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:08:35.052837+0000","last_undegraded":"2024-01-11T10:08:35.052837+0000","last_fullsized":"2024-01-11T10:08:35.052837+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"17'1152","reported_seq":1165,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.031816+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:09:58.031816+0000","last_peered":"2024-01-11T10:09:58.031816+0000","last_clean":"2024-01-11T10:09:58.031816+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:09:58.031816+0000","last_undegraded":"2024-01-11T10:09:58.031816+0000","last_fullsized":"2024-01-11T10:09:58.031816+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":1152,"num_object_clones":0,"num_object_copies":3456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1152,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"18'1030","reported_seq":1044,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.036800+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:09:57.036800+0000","last_peered":"2024-01-11T10:09:57.036800+0000","last_clean":"2024-01-11T10:09:57.036800+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:09:57.036800+0000","last_undegraded":"2024-01-11T10:09:57.036800+0000","last_fullsized":"2024-01-11T10:09:57.036800+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.053590+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:09:58.053590+0000","last_peered":"2024-01-11T10:09:58.053590+0000","last_clean":"2024-01-11T10:09:58.053590+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:09:58.053590+0000","last_undegraded":"2024-01-11T10:09:58.053590+0000","last_fullsized":"2024-01-11T10:09:58.053590+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"17'1097","reported_seq":1111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.049945+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:09:58.049945+0000","last_peered":"2024-01-11T10:09:58.049945+0000","last_clean":"2024-01-11T10:09:58.049945+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:09:58.049945+0000","last_undegraded":"2024-01-11T10:09:58.049945+0000","last_fullsized":"2024-01-11T10:09:58.049945+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1098,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":17,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:08:35.076093+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:08:35.076093+0000","last_peered":"2024-01-11T10:08:35.076093+0000","last_clean":"2024-01-11T10:08:35.076093+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:08:35.076093+0000","last_undegraded":"2024-01-11T10:08:35.076093+0000","last_fullsized":"2024-01-11T10:08:35.076093+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"17'1073","reported_seq":1085,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.056316+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:09:57.056316+0000","last_peered":"2024-01-11T10:09:57.056316+0000","last_clean":"2024-01-11T10:09:57.056316+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:09:57.056316+0000","last_undegraded":"2024-01-11T10:09:57.056316+0000","last_fullsized":"2024-01-11T10:09:57.056316+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1073,"log_dups_size":0,"ondisk_log_size":1073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70385664,"num_objects":1074,"num_object_clones":0,"num_object_copies":3222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1074,"num_write_kb":68736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.050593+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:09:58.050593+0000","last_peered":"2024-01-11T10:09:58.050593+0000","last_clean":"2024-01-11T10:09:58.050593+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:09:58.050593+0000","last_undegraded":"2024-01-11T10:09:58.050593+0000","last_fullsized":"2024-01-11T10:09:58.050593+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"17'1037","reported_seq":1051,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.019884+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:09:58.019884+0000","last_peered":"2024-01-11T10:09:58.019884+0000","last_clean":"2024-01-11T10:09:58.019884+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:09:58.019884+0000","last_undegraded":"2024-01-11T10:09:58.019884+0000","last_fullsized":"2024-01-11T10:09:58.019884+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68026368,"num_objects":1038,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":66432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-01-11T10:09:57.022115+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:09:57.022115+0000","last_peered":"2024-01-11T10:09:57.022115+0000","last_clean":"2024-01-11T10:09:57.022115+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:09:57.022115+0000","last_undegraded":"2024-01-11T10:09:57.022115+0000","last_fullsized":"2024-01-11T10:09:57.022115+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1119748096,"num_objects":17086,"num_object_clones":0,"num_object_copies":51258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17086,"num_write_kb":1093504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17072,"ondisk_log_size":17072,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86128,"kb_used_data":46824,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285712,"statfs":{"total":96636764160,"available":96548569088,"internally_reserved":0,"allocated":47947776,"data_stored":764519189,"data_compressed":36302709,"data_compressed_allocated":47783936,"data_compressed_original":764461056,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705764,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":86700,"kb_used_data":47396,"kb_used_omap":0,"kb_used_meta":39296,"kb_avail":94285140,"statfs":{"total":96636764160,"available":96547983360,"internally_reserved":0,"allocated":48533504,"data_stored":773890837,"data_compressed":36747686,"data_compressed_allocated":48369664,"data_compressed_original":773832704,"omap_allocated":0,"internal_metadata":40239104},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112212,"kb_used_data":59020,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94259628,"statfs":{"total":96636764160,"available":96521859072,"internally_reserved":0,"allocated":60436480,"data_stored":964338453,"data_compressed":45790934,"data_compressed_allocated":60272640,"data_compressed_original":964280320,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705763,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95356,"kb_used_data":50484,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94276484,"statfs":{"total":96636764160,"available":96539119616,"internally_reserved":0,"allocated":51695616,"data_stored":824877301,"data_compressed":39166791,"data_compressed_allocated":51552256,"data_compressed_original":824836096,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:10:02.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 87 16 17911 17895 12.8545 12.25 0.0460145 0.0777373 2024-01-11T10:10:03.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 88 16 18097 18081 12.8406 11.625 0.10671 0.0778278 2024-01-11T10:10:03.449 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:03.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:03.595+0000 7f4a422ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:03.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:03.599+0000 7f4a422ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:04.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 89 16 18279 18263 12.8241 11.375 0.11373 0.0779257 2024-01-11T10:10:05.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 90 13 18470 18457 12.8163 12.125 0.0954222 0.0779936 2024-01-11T10:10:05.833 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:05.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:05.966+0000 7fab23d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:05.954 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:05.970+0000 7fab23d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:06.028 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Total time run: 90.0353 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Total writes made: 18470 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Write size: 65536 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object size: 65536 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Bandwidth (MB/sec): 12.8214 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev Bandwidth: 0.872902 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max bandwidth (MB/sec): 14.9375 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min bandwidth (MB/sec): 10.5625 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Average IOPS: 205 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev IOPS: 13.9664 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max IOPS: 239 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min IOPS: 169 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Average Latency(s): 0.0779773 2024-01-11T10:10:06.029 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev Latency(s): 0.0321953 2024-01-11T10:10:06.030 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max latency(s): 0.217041 2024-01-11T10:10:06.030 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min latency(s): 0.0112295 2024-01-11T10:10:06.053 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Cleaning up (deleting benchmark objects) 2024-01-11T10:10:06.245 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:06.246 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:06.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:10:06.364 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:06.376+0000 7f96fc26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:06.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:06.384+0000 7f96fc26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:06.731 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:10:06.732 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:10:06.744 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-01-11T10:10:06.115758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1181745696,"num_objects":18025,"num_object_clones":0,"num_object_copies":54075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":18160,"num_write_kb":1155366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18104,"ondisk_log_size":18104,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":411168,"kb_used_data":215040,"kb_used_omap":0,"kb_used_meta":196096,"kb_avail":377076192,"statfs":{"total":386547056640,"available":386126020608,"internally_reserved":0,"allocated":220200960,"data_stored":3511365373,"data_compressed":166731056,"data_compressed_allocated":219459584,"data_compressed_original":3511107584,"omap_allocated":0,"internal_metadata":200802304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":93061120,"num_objects":1420,"num_object_clones":0,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1420,"num_write_kb":90880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001122"},"pg_stats":[{"pgid":"3.f","version":"21'1110","reported_seq":1126,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:03.096298+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:10:03.096298+0000","last_peered":"2024-01-11T10:10:03.096298+0000","last_clean":"2024-01-11T10:10:03.096298+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:10:03.096298+0000","last_undegraded":"2024-01-11T10:10:03.096298+0000","last_fullsized":"2024-01-11T10:10:03.096298+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1110,"log_dups_size":0,"ondisk_log_size":1110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72810496,"num_objects":1111,"num_object_clones":0,"num_object_copies":3333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1111,"num_write_kb":71104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'1152","reported_seq":1168,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:03.030963+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:10:03.030963+0000","last_peered":"2024-01-11T10:10:03.030963+0000","last_clean":"2024-01-11T10:10:03.030963+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:10:03.030963+0000","last_undegraded":"2024-01-11T10:10:03.030963+0000","last_fullsized":"2024-01-11T10:10:03.030963+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1152,"log_dups_size":0,"ondisk_log_size":1152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75563008,"num_objects":1153,"num_object_clones":0,"num_object_copies":3459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1153,"num_write_kb":73792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"19'1093","reported_seq":1109,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:10:01.824819+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:10:01.824819+0000","last_peered":"2024-01-11T10:10:01.824819+0000","last_clean":"2024-01-11T10:10:01.824819+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:10:01.824819+0000","last_undegraded":"2024-01-11T10:10:01.824819+0000","last_fullsized":"2024-01-11T10:10:01.824819+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1093,"log_dups_size":0,"ondisk_log_size":1093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71696384,"num_objects":1094,"num_object_clones":0,"num_object_copies":3282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1094,"num_write_kb":70016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'1119","reported_seq":1135,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:03.106354+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:10:03.106354+0000","last_peered":"2024-01-11T10:10:03.106354+0000","last_clean":"2024-01-11T10:10:03.106354+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:10:03.106354+0000","last_undegraded":"2024-01-11T10:10:03.106354+0000","last_fullsized":"2024-01-11T10:10:03.106354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1119,"log_dups_size":0,"ondisk_log_size":1119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1120,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"20'1074","reported_seq":1092,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.071684+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:10:02.071684+0000","last_peered":"2024-01-11T10:10:02.071684+0000","last_clean":"2024-01-11T10:10:02.071684+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:10:02.071684+0000","last_undegraded":"2024-01-11T10:10:02.071684+0000","last_fullsized":"2024-01-11T10:10:02.071684+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70451200,"num_objects":1075,"num_object_clones":0,"num_object_copies":3225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1075,"num_write_kb":68800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'1132","reported_seq":1148,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.964389+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:10:02.964389+0000","last_peered":"2024-01-11T10:10:02.964389+0000","last_clean":"2024-01-11T10:10:02.964389+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:10:02.964389+0000","last_undegraded":"2024-01-11T10:10:02.964389+0000","last_fullsized":"2024-01-11T10:10:02.964389+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1132,"log_dups_size":0,"ondisk_log_size":1132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74252288,"num_objects":1133,"num_object_clones":0,"num_object_copies":3399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1133,"num_write_kb":72512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'1144","reported_seq":1160,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:03.066660+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:10:03.066660+0000","last_peered":"2024-01-11T10:10:03.066660+0000","last_clean":"2024-01-11T10:10:03.066660+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:10:03.066660+0000","last_undegraded":"2024-01-11T10:10:03.066660+0000","last_fullsized":"2024-01-11T10:10:03.066660+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75038720,"num_objects":1145,"num_object_clones":0,"num_object_copies":3435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1145,"num_write_kb":73280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"19'1115","reported_seq":1131,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.007206+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:10:02.007206+0000","last_peered":"2024-01-11T10:10:02.007206+0000","last_clean":"2024-01-11T10:10:02.007206+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:10:02.007206+0000","last_undegraded":"2024-01-11T10:10:02.007206+0000","last_fullsized":"2024-01-11T10:10:02.007206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1115,"log_dups_size":0,"ondisk_log_size":1115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73138176,"num_objects":1116,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1116,"num_write_kb":71424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":154,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T10:09:59.058954+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:09:59.058954+0000","last_peered":"2024-01-11T10:09:59.058954+0000","last_clean":"2024-01-11T10:09:59.058954+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:09:59.058954+0000","last_undegraded":"2024-01-11T10:09:59.058954+0000","last_fullsized":"2024-01-11T10:09:59.058954+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.222714+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:09:58.222714+0000","last_peered":"2024-01-11T10:09:58.222714+0000","last_clean":"2024-01-11T10:09:58.222714+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:09:58.222714+0000","last_undegraded":"2024-01-11T10:09:58.222714+0000","last_fullsized":"2024-01-11T10:09:58.222714+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"19'1157","reported_seq":1173,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.032672+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:10:02.032672+0000","last_peered":"2024-01-11T10:10:02.032672+0000","last_clean":"2024-01-11T10:10:02.032672+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:10:02.032672+0000","last_undegraded":"2024-01-11T10:10:02.032672+0000","last_fullsized":"2024-01-11T10:10:02.032672+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75890688,"num_objects":1158,"num_object_clones":0,"num_object_copies":3474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1158,"num_write_kb":74112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"20'1163","reported_seq":1181,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T10:10:01.877669+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:10:01.877669+0000","last_peered":"2024-01-11T10:10:01.877669+0000","last_clean":"2024-01-11T10:10:01.877669+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:10:01.877669+0000","last_undegraded":"2024-01-11T10:10:01.877669+0000","last_fullsized":"2024-01-11T10:10:01.877669+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76283904,"num_objects":1164,"num_object_clones":0,"num_object_copies":3492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1164,"num_write_kb":74496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.217491+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:09:58.217491+0000","last_peered":"2024-01-11T10:09:58.217491+0000","last_clean":"2024-01-11T10:09:58.217491+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:09:58.217491+0000","last_undegraded":"2024-01-11T10:09:58.217491+0000","last_fullsized":"2024-01-11T10:09:58.217491+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"20'1072","reported_seq":1090,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.055417+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:10:02.055417+0000","last_peered":"2024-01-11T10:10:02.055417+0000","last_clean":"2024-01-11T10:10:02.055417+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:10:02.055417+0000","last_undegraded":"2024-01-11T10:10:02.055417+0000","last_fullsized":"2024-01-11T10:10:02.055417+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70320128,"num_objects":1073,"num_object_clones":0,"num_object_copies":3219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1073,"num_write_kb":68672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'1213","reported_seq":1229,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:03.106551+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:10:03.106551+0000","last_peered":"2024-01-11T10:10:03.106551+0000","last_clean":"2024-01-11T10:10:03.106551+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:10:03.106551+0000","last_undegraded":"2024-01-11T10:10:03.106551+0000","last_fullsized":"2024-01-11T10:10:03.106551+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79560704,"num_objects":1214,"num_object_clones":0,"num_object_copies":3642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":77696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"19'1083","reported_seq":1099,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.041160+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:10:02.041160+0000","last_peered":"2024-01-11T10:10:02.041160+0000","last_clean":"2024-01-11T10:10:02.041160+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:10:02.041160+0000","last_undegraded":"2024-01-11T10:10:02.041160+0000","last_fullsized":"2024-01-11T10:10:02.041160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1083,"log_dups_size":0,"ondisk_log_size":1083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":69376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'1154","reported_seq":1170,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.940217+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:10:02.940217+0000","last_peered":"2024-01-11T10:10:02.940217+0000","last_clean":"2024-01-11T10:10:02.940217+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:10:02.940217+0000","last_undegraded":"2024-01-11T10:10:02.940217+0000","last_fullsized":"2024-01-11T10:10:02.940217+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1154,"log_dups_size":0,"ondisk_log_size":1154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75694080,"num_objects":1155,"num_object_clones":0,"num_object_copies":3465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1155,"num_write_kb":73920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T10:09:59.048426+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:09:59.048426+0000","last_peered":"2024-01-11T10:09:59.048426+0000","last_clean":"2024-01-11T10:09:59.048426+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:09:59.048426+0000","last_undegraded":"2024-01-11T10:09:59.048426+0000","last_fullsized":"2024-01-11T10:09:59.048426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"20'1140","reported_seq":1158,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.107506+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:10:02.107506+0000","last_peered":"2024-01-11T10:10:02.107506+0000","last_clean":"2024-01-11T10:10:02.107506+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:10:02.107506+0000","last_undegraded":"2024-01-11T10:10:02.107506+0000","last_fullsized":"2024-01-11T10:10:02.107506+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1140,"log_dups_size":0,"ondisk_log_size":1140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776576,"num_objects":1141,"num_object_clones":0,"num_object_copies":3423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1141,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'1086","reported_seq":1102,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.979508+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:10:02.979508+0000","last_peered":"2024-01-11T10:10:02.979508+0000","last_clean":"2024-01-11T10:10:02.979508+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:10:02.979508+0000","last_undegraded":"2024-01-11T10:10:02.979508+0000","last_fullsized":"2024-01-11T10:10:02.979508+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1086,"log_dups_size":0,"ondisk_log_size":1086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71237632,"num_objects":1087,"num_object_clones":0,"num_object_copies":3261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1087,"num_write_kb":69568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-01-11T10:09:58.216061+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:09:58.216061+0000","last_peered":"2024-01-11T10:09:58.216061+0000","last_clean":"2024-01-11T10:09:58.216061+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:09:58.216061+0000","last_undegraded":"2024-01-11T10:09:58.216061+0000","last_fullsized":"2024-01-11T10:09:58.216061+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1181155328,"num_objects":18023,"num_object_clones":0,"num_object_copies":54069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18023,"num_write_kb":1153472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18007,"ondisk_log_size":18007,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94244,"kb_used_data":49372,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94277596,"statfs":{"total":96636764160,"available":96540258304,"internally_reserved":0,"allocated":50556928,"data_stored":805754313,"data_compressed":38260157,"data_compressed_allocated":50360320,"data_compressed_original":805683200,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705769,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94904,"kb_used_data":50032,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94276936,"statfs":{"total":96636764160,"available":96539582464,"internally_reserved":0,"allocated":51232768,"data_stored":816567753,"data_compressed":38773598,"data_compressed_allocated":51036160,"data_compressed_original":816496640,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":119724,"kb_used_data":62372,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94252116,"statfs":{"total":96636764160,"available":96514166784,"internally_reserved":0,"allocated":63868928,"data_stored":1018874140,"data_compressed":48380118,"data_compressed_allocated":63680512,"data_compressed_original":1018806272,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705768,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":102296,"kb_used_data":53264,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94269544,"statfs":{"total":96636764160,"available":96532013056,"internally_reserved":0,"allocated":54542336,"data_stored":870169167,"data_compressed":41317183,"data_compressed_allocated":54382592,"data_compressed_original":870121472,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:10:08.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:08.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:08.174+0000 7f4f7279e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:08.163 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:08.179+0000 7f4f7279e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:10.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:10.512 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:10.527+0000 7fdc21b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:10.516 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:10.532+0000 7fdc21b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:11.746 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:11.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:11.747 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:11.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:11.892 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:11.904+0000 7efc994de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:11.896 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:11.912+0000 7efc994de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:11.926 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:11.926 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:11.927 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:11.929 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:11.929 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:12.069 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.081+0000 7f8b14e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.073 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.089+0000 7f8b14e13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:12.087 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:12.088 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:12.090 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:12.090 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:12.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.217+0000 7f307466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.211 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.228+0000 7f307466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:12.225 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:12.226 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:12.228 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:12.229 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:12.349 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.361+0000 7f658e86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.353 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.369+0000 7f658e86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.423 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:12.423 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:12.424 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:12.427 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:12.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:12.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.912+0000 7f6b0dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:12.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:12.917+0000 7f6b0dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:15.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:15.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:15.287+0000 7fe5bfa17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:15.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:15.293+0000 7fe5bfa17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:17.428 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:17.428 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:17.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:10:17.463 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:17.551 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:17.563+0000 7f284582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:17.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:17.572+0000 7f284582d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:17.617 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:17.631+0000 7f633ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:17.620 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:17.637+0000 7f633ad5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:17.771 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:10:17.771 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:10:17.784 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-01-11T10:10:16.117396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1123680824,"num_objects":17140,"num_object_clones":0,"num_object_copies":51420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":19941,"num_write_kb":1183975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19885,"ondisk_log_size":19885,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":408936,"kb_used_data":208640,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":377078424,"statfs":{"total":386547056640,"available":386128306176,"internally_reserved":0,"allocated":213647360,"data_stored":3405927756,"data_compressed":161727178,"data_compressed_allocated":212869120,"data_compressed_original":3405660160,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58064872,"num_objects":-885,"num_object_clones":0,"num_object_copies":-2655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1781,"num_write_kb":28609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002000"},"pg_stats":[{"pgid":"3.f","version":"21'1229","reported_seq":1245,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:13.020805+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:10:13.020805+0000","last_peered":"2024-01-11T10:10:13.020805+0000","last_clean":"2024-01-11T10:10:13.020805+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:10:13.020805+0000","last_undegraded":"2024-01-11T10:10:13.020805+0000","last_fullsized":"2024-01-11T10:10:13.020805+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1229,"log_dups_size":0,"ondisk_log_size":1229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67371008,"num_objects":1028,"num_object_clones":0,"num_object_copies":3084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1230,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'1261","reported_seq":1277,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.684611+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:10:12.684611+0000","last_peered":"2024-01-11T10:10:12.684611+0000","last_clean":"2024-01-11T10:10:12.684611+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:10:12.684611+0000","last_undegraded":"2024-01-11T10:10:12.684611+0000","last_fullsized":"2024-01-11T10:10:12.684611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71565312,"num_objects":1092,"num_object_clones":0,"num_object_copies":3276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1262,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1198","reported_seq":1217,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.075160+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:10:12.075160+0000","last_peered":"2024-01-11T10:10:12.075160+0000","last_clean":"2024-01-11T10:10:12.075160+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:10:12.075160+0000","last_undegraded":"2024-01-11T10:10:12.075160+0000","last_fullsized":"2024-01-11T10:10:12.075160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1198,"log_dups_size":0,"ondisk_log_size":1198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69074968,"num_objects":1055,"num_object_clones":0,"num_object_copies":3165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1199,"num_write_kb":72065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'1242","reported_seq":1258,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:13.059041+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:10:13.059041+0000","last_peered":"2024-01-11T10:10:13.059041+0000","last_clean":"2024-01-11T10:10:13.059041+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:10:13.059041+0000","last_undegraded":"2024-01-11T10:10:13.059041+0000","last_fullsized":"2024-01-11T10:10:13.059041+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1242,"log_dups_size":0,"ondisk_log_size":1242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68222976,"num_objects":1041,"num_object_clones":0,"num_object_copies":3123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1243,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'1178","reported_seq":1198,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.066522+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:10:12.066522+0000","last_peered":"2024-01-11T10:10:12.066522+0000","last_clean":"2024-01-11T10:10:12.066522+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:10:12.066522+0000","last_undegraded":"2024-01-11T10:10:12.066522+0000","last_fullsized":"2024-01-11T10:10:12.066522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1178,"log_dups_size":0,"ondisk_log_size":1178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67960832,"num_objects":1037,"num_object_clones":0,"num_object_copies":3111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1179,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'1235","reported_seq":1251,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.983065+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:10:12.983065+0000","last_peered":"2024-01-11T10:10:12.983065+0000","last_clean":"2024-01-11T10:10:12.983065+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:10:12.983065+0000","last_undegraded":"2024-01-11T10:10:12.983065+0000","last_fullsized":"2024-01-11T10:10:12.983065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1235,"log_dups_size":0,"ondisk_log_size":1235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70909952,"num_objects":1082,"num_object_clones":0,"num_object_copies":3246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'1241","reported_seq":1257,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:13.073510+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:10:13.073510+0000","last_peered":"2024-01-11T10:10:13.073510+0000","last_clean":"2024-01-11T10:10:13.073510+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:10:13.073510+0000","last_undegraded":"2024-01-11T10:10:13.073510+0000","last_fullsized":"2024-01-11T10:10:13.073510+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1242,"num_write_kb":74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'1202","reported_seq":1220,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:11.797930+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:10:11.797930+0000","last_peered":"2024-01-11T10:10:11.797930+0000","last_clean":"2024-01-11T10:10:11.797930+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:10:11.797930+0000","last_undegraded":"2024-01-11T10:10:11.797930+0000","last_fullsized":"2024-01-11T10:10:11.797930+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1202,"log_dups_size":0,"ondisk_log_size":1202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71499776,"num_objects":1091,"num_object_clones":0,"num_object_copies":3273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":156,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.205924+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:10:02.205924+0000","last_peered":"2024-01-11T10:10:02.205924+0000","last_clean":"2024-01-11T10:10:02.205924+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:10:02.205924+0000","last_undegraded":"2024-01-11T10:10:02.205924+0000","last_fullsized":"2024-01-11T10:10:02.205924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.182609+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:10:02.182609+0000","last_peered":"2024-01-11T10:10:02.182609+0000","last_clean":"2024-01-11T10:10:02.182609+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:10:02.182609+0000","last_undegraded":"2024-01-11T10:10:02.182609+0000","last_fullsized":"2024-01-11T10:10:02.182609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1263","reported_seq":1281,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.065060+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:10:12.065060+0000","last_peered":"2024-01-11T10:10:12.065060+0000","last_clean":"2024-01-11T10:10:12.065060+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:10:12.065060+0000","last_undegraded":"2024-01-11T10:10:12.065060+0000","last_fullsized":"2024-01-11T10:10:12.065060+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1263,"log_dups_size":0,"ondisk_log_size":1263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73269248,"num_objects":1118,"num_object_clones":0,"num_object_copies":3354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1264,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'1290","reported_seq":1310,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.031036+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:10:12.031036+0000","last_peered":"2024-01-11T10:10:12.031036+0000","last_clean":"2024-01-11T10:10:12.031036+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:10:12.031036+0000","last_undegraded":"2024-01-11T10:10:12.031036+0000","last_fullsized":"2024-01-11T10:10:12.031036+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1290,"log_dups_size":0,"ondisk_log_size":1290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1291,"num_write_kb":76480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.179920+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:10:02.179920+0000","last_peered":"2024-01-11T10:10:02.179920+0000","last_clean":"2024-01-11T10:10:02.179920+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:10:02.179920+0000","last_undegraded":"2024-01-11T10:10:02.179920+0000","last_fullsized":"2024-01-11T10:10:02.179920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'1184","reported_seq":1204,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:11.806395+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:10:11.806395+0000","last_peered":"2024-01-11T10:10:11.806395+0000","last_clean":"2024-01-11T10:10:11.806395+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:10:11.806395+0000","last_undegraded":"2024-01-11T10:10:11.806395+0000","last_fullsized":"2024-01-11T10:10:11.806395+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68354048,"num_objects":1043,"num_object_clones":0,"num_object_copies":3129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1185,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'1328","reported_seq":1344,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:13.045110+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:10:13.045110+0000","last_peered":"2024-01-11T10:10:13.045110+0000","last_clean":"2024-01-11T10:10:13.045110+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:10:13.045110+0000","last_undegraded":"2024-01-11T10:10:13.045110+0000","last_fullsized":"2024-01-11T10:10:13.045110+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1328,"log_dups_size":0,"ondisk_log_size":1328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74776576,"num_objects":1141,"num_object_clones":0,"num_object_copies":3423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1329,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'1208","reported_seq":1226,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.127771+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:10:12.127771+0000","last_peered":"2024-01-11T10:10:12.127771+0000","last_clean":"2024-01-11T10:10:12.127771+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:10:12.127771+0000","last_undegraded":"2024-01-11T10:10:12.127771+0000","last_fullsized":"2024-01-11T10:10:12.127771+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1208,"log_dups_size":0,"ondisk_log_size":1208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67436544,"num_objects":1029,"num_object_clones":0,"num_object_copies":3087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1209,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'1250","reported_seq":1266,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:13.071121+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:10:13.071121+0000","last_peered":"2024-01-11T10:10:13.071121+0000","last_clean":"2024-01-11T10:10:13.071121+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:10:13.071121+0000","last_undegraded":"2024-01-11T10:10:13.071121+0000","last_fullsized":"2024-01-11T10:10:13.071121+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71499776,"num_objects":1091,"num_object_clones":0,"num_object_copies":3273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1251,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204927+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:10:02.204927+0000","last_peered":"2024-01-11T10:10:02.204927+0000","last_clean":"2024-01-11T10:10:02.204927+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:10:02.204927+0000","last_undegraded":"2024-01-11T10:10:02.204927+0000","last_fullsized":"2024-01-11T10:10:02.204927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'1275","reported_seq":1295,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.114762+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:10:12.114762+0000","last_peered":"2024-01-11T10:10:12.114762+0000","last_clean":"2024-01-11T10:10:12.114762+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:10:12.114762+0000","last_undegraded":"2024-01-11T10:10:12.114762+0000","last_fullsized":"2024-01-11T10:10:12.114762+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1275,"log_dups_size":0,"ondisk_log_size":1275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1276,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'1204","reported_seq":1220,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:12.944043+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:10:12.944043+0000","last_peered":"2024-01-11T10:10:12.944043+0000","last_clean":"2024-01-11T10:10:12.944043+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:10:12.944043+0000","last_undegraded":"2024-01-11T10:10:12.944043+0000","last_fullsized":"2024-01-11T10:10:12.944043+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66125824,"num_objects":1009,"num_object_clones":0,"num_object_copies":3027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.196049+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:10:02.196049+0000","last_peered":"2024-01-11T10:10:02.196049+0000","last_clean":"2024-01-11T10:10:02.196049+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:10:02.196049+0000","last_undegraded":"2024-01-11T10:10:02.196049+0000","last_fullsized":"2024-01-11T10:10:02.196049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":1123090456,"num_objects":17138,"num_object_clones":0,"num_object_copies":51414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19804,"num_write_kb":1182081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19788,"ondisk_log_size":19788,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96472,"kb_used_data":47440,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94275368,"statfs":{"total":96636764160,"available":96537976832,"internally_reserved":0,"allocated":48578560,"data_stored":774100425,"data_compressed":36757791,"data_compressed_allocated":48381952,"data_compressed_original":774029312,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705779,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97000,"kb_used_data":47964,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94274840,"statfs":{"total":96636764160,"available":96537436160,"internally_reserved":0,"allocated":49115136,"data_stored":782620129,"data_compressed":37162350,"data_compressed_allocated":48914432,"data_compressed_original":782548992,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":110056,"kb_used_data":61024,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94261784,"statfs":{"total":96636764160,"available":96524066816,"internally_reserved":0,"allocated":62488576,"data_stored":996595169,"data_compressed":47323030,"data_compressed_allocated":62287872,"data_compressed_original":996524032,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705778,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":105408,"kb_used_data":52212,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94266432,"statfs":{"total":96636764160,"available":96528826368,"internally_reserved":0,"allocated":53465088,"data_stored":852612033,"data_compressed":40484007,"data_compressed_allocated":53284864,"data_compressed_original":852557824,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:10:19.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:19.994 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:20.008+0000 7f5103220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:19.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:20.014+0000 7f5103220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:22.224 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:22.340 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:22.355+0000 7fc8546ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:22.344 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:22.360+0000 7fc8546ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:22.786 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:22.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:22.786 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:22.787 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:22.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:22.913+0000 7fe900864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:22.905 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:22.921+0000 7fe900864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:22.949 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:22.949 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:22.949 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:22.949 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:22.950 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:22.953 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:22.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:23.083 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:23.094+0000 7f22f835e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:23.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:23.103+0000 7f22f835e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:23.113 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:23.114 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:23.117 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:23.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:23.271 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:23.283+0000 7f1877a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:23.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:23.292+0000 7f1877a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:23.327 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:23.327 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:23.327 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:23.327 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:23.327 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:23.330 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:23.330 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:23.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:23.456+0000 7fad94e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:23.453 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:23.470+0000 7fad94e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:23.524 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:23.524 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:23.525 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:23.528 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:24.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:24.746 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:24.761+0000 7fd27599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:24.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:24.766+0000 7fd27599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:26.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:27.081 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:27.095+0000 7fb0f6b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:27.084 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:27.100+0000 7fb0f6b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:28.529 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:28.529 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:28.529 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:28.530 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:28.671 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:28.680+0000 7f96b4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:28.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:28.690+0000 7f96b4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:28.707 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:28.707 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:28.707 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:28.708 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:28.710 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:28.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:28.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:28.839+0000 7f1e5382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:28.832 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:28.848+0000 7f1e5382d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:28.874 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:28.875 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:28.877 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:28.877 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:28.997 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:29.005+0000 7f94fd22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:29.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:29.016+0000 7f94fd22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:29.091 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:29.092 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:29.095 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:29.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:29.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:29.239+0000 7f760946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:29.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:29.249+0000 7f760946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:29.252 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:29.252 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:29.252 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:29.253 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:29.254 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:29.255 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:29.255 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:29.361 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:29.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:29.488+0000 7fcf0de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:29.477 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:29.493+0000 7fcf0de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:31.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:31.888+0000 7f5eaf017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:31.894+0000 7f5eaf017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.102 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:34.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.231+0000 7f66c0a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.236+0000 7f66c0a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.256 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:34.256 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:34.256 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:34.256 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:34.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.389+0000 7f70e1cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.401+0000 7f70e1cf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:34.442 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:34.443 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:34.443 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:34.443 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:34.443 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:34.443 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:34.445 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:34.445 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:34.445 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:34.562 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.571+0000 7f303819e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.566 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.581+0000 7f303819e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:34.597 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:34.598 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:34.600 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:34.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:34.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.729+0000 7f12eacad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.737+0000 7f12eacad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.739 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:34.739 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:34.739 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:34.739 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:34.739 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:34.739 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:34.740 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:34.743 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:34.743 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:34.887 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.898+0000 7fb562b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:34.906+0000 7fb562b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:34.974 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:34.977 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:36.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:36.579 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:36.593+0000 7f6472d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:36.583 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:36.599+0000 7f6472d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:38.895 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:39.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:39.025+0000 7ffba926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:39.015 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:39.030+0000 7ffba926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:39.978 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:39.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:39.979 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:10:40.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:40.120+0000 7fc225f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:40.113 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:40.129+0000 7fc225f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:40.408 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:10:40.408 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:10:40.420 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-01-11T10:10:40.121696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":774832696,"num_objects":11817,"num_object_clones":0,"num_object_copies":35451,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":25264,"num_write_kb":1183975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25208,"ondisk_log_size":25208,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":345076,"kb_used_data":144784,"kb_used_omap":0,"kb_used_meta":200256,"kb_avail":377142284,"statfs":{"total":386547056640,"available":386193698816,"internally_reserved":0,"allocated":148258816,"data_stored":2359711052,"data_compressed":112047210,"data_compressed_allocated":147480576,"data_compressed_original":2359443456,"omap_allocated":0,"internal_metadata":205062144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-170590208,"num_objects":-2603,"num_object_clones":0,"num_object_copies":-7809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2603,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002305"},"pg_stats":[{"pgid":"3.f","version":"21'1550","reported_seq":1566,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:38.081765+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:10:38.081765+0000","last_peered":"2024-01-11T10:10:38.081765+0000","last_clean":"2024-01-11T10:10:38.081765+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:10:38.081765+0000","last_undegraded":"2024-01-11T10:10:38.081765+0000","last_fullsized":"2024-01-11T10:10:38.081765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1550,"log_dups_size":0,"ondisk_log_size":1550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46333952,"num_objects":707,"num_object_clones":0,"num_object_copies":2121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1551,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'1593","reported_seq":1609,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:38.091184+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:10:38.091184+0000","last_peered":"2024-01-11T10:10:38.091184+0000","last_clean":"2024-01-11T10:10:38.091184+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:10:38.091184+0000","last_undegraded":"2024-01-11T10:10:38.091184+0000","last_fullsized":"2024-01-11T10:10:38.091184+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1593,"log_dups_size":0,"ondisk_log_size":1593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49807360,"num_objects":760,"num_object_clones":0,"num_object_copies":2280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1594,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'1518","reported_seq":1537,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.108549+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:10:37.108549+0000","last_peered":"2024-01-11T10:10:37.108549+0000","last_clean":"2024-01-11T10:10:37.108549+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:10:37.108549+0000","last_undegraded":"2024-01-11T10:10:37.108549+0000","last_fullsized":"2024-01-11T10:10:37.108549+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103448,"num_objects":735,"num_object_clones":0,"num_object_copies":2205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1519,"num_write_kb":72065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'1555","reported_seq":1571,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:38.065048+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:10:38.065048+0000","last_peered":"2024-01-11T10:10:38.065048+0000","last_clean":"2024-01-11T10:10:38.065048+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:10:38.065048+0000","last_undegraded":"2024-01-11T10:10:38.065048+0000","last_fullsized":"2024-01-11T10:10:38.065048+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1555,"log_dups_size":0,"ondisk_log_size":1555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1556,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'1507","reported_seq":1527,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.066065+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:10:37.066065+0000","last_peered":"2024-01-11T10:10:37.066065+0000","last_clean":"2024-01-11T10:10:37.066065+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:10:37.066065+0000","last_undegraded":"2024-01-11T10:10:37.066065+0000","last_fullsized":"2024-01-11T10:10:37.066065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1507,"log_dups_size":0,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46399488,"num_objects":708,"num_object_clones":0,"num_object_copies":2124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1508,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'1594","reported_seq":1610,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:38.176902+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:10:38.176902+0000","last_peered":"2024-01-11T10:10:38.176902+0000","last_clean":"2024-01-11T10:10:38.176902+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:10:38.176902+0000","last_undegraded":"2024-01-11T10:10:38.176902+0000","last_fullsized":"2024-01-11T10:10:38.176902+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1594,"log_dups_size":0,"ondisk_log_size":1594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47382528,"num_objects":723,"num_object_clones":0,"num_object_copies":2169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1595,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'1572","reported_seq":1588,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:38.039065+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:10:38.039065+0000","last_peered":"2024-01-11T10:10:38.039065+0000","last_clean":"2024-01-11T10:10:38.039065+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:10:38.039065+0000","last_undegraded":"2024-01-11T10:10:38.039065+0000","last_fullsized":"2024-01-11T10:10:38.039065+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1572,"log_dups_size":0,"ondisk_log_size":1572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49348608,"num_objects":753,"num_object_clones":0,"num_object_copies":2259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1573,"num_write_kb":74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'1528","reported_seq":1546,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.105886+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:10:37.105886+0000","last_peered":"2024-01-11T10:10:37.105886+0000","last_clean":"2024-01-11T10:10:37.105886+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:10:37.105886+0000","last_undegraded":"2024-01-11T10:10:37.105886+0000","last_fullsized":"2024-01-11T10:10:37.105886+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1528,"log_dups_size":0,"ondisk_log_size":1528,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1529,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":156,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.205924+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:10:02.205924+0000","last_peered":"2024-01-11T10:10:02.205924+0000","last_clean":"2024-01-11T10:10:02.205924+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:10:02.205924+0000","last_undegraded":"2024-01-11T10:10:02.205924+0000","last_fullsized":"2024-01-11T10:10:02.205924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.182609+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:10:02.182609+0000","last_peered":"2024-01-11T10:10:02.182609+0000","last_clean":"2024-01-11T10:10:02.182609+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:10:02.182609+0000","last_undegraded":"2024-01-11T10:10:02.182609+0000","last_fullsized":"2024-01-11T10:10:02.182609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'1631","reported_seq":1649,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.028367+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:10:37.028367+0000","last_peered":"2024-01-11T10:10:37.028367+0000","last_clean":"2024-01-11T10:10:37.028367+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:10:37.028367+0000","last_undegraded":"2024-01-11T10:10:37.028367+0000","last_fullsized":"2024-01-11T10:10:37.028367+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1631,"log_dups_size":0,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49152000,"num_objects":750,"num_object_clones":0,"num_object_copies":2250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1632,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'1624","reported_seq":1644,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:36.969198+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:10:36.969198+0000","last_peered":"2024-01-11T10:10:36.969198+0000","last_clean":"2024-01-11T10:10:36.969198+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:10:36.969198+0000","last_undegraded":"2024-01-11T10:10:36.969198+0000","last_fullsized":"2024-01-11T10:10:36.969198+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1625,"num_write_kb":76480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.179920+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:10:02.179920+0000","last_peered":"2024-01-11T10:10:02.179920+0000","last_clean":"2024-01-11T10:10:02.179920+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:10:02.179920+0000","last_undegraded":"2024-01-11T10:10:02.179920+0000","last_fullsized":"2024-01-11T10:10:02.179920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'1515","reported_seq":1535,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.118188+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:10:37.118188+0000","last_peered":"2024-01-11T10:10:37.118188+0000","last_clean":"2024-01-11T10:10:37.118188+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:10:37.118188+0000","last_undegraded":"2024-01-11T10:10:37.118188+0000","last_fullsized":"2024-01-11T10:10:37.118188+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1516,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'1681","reported_seq":1697,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.904698+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:10:37.904698+0000","last_peered":"2024-01-11T10:10:37.904698+0000","last_clean":"2024-01-11T10:10:37.904698+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:10:37.904698+0000","last_undegraded":"2024-01-11T10:10:37.904698+0000","last_fullsized":"2024-01-11T10:10:37.904698+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1681,"log_dups_size":0,"ondisk_log_size":1681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1682,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'1540","reported_seq":1558,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:36.903526+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:10:36.903526+0000","last_peered":"2024-01-11T10:10:36.903526+0000","last_clean":"2024-01-11T10:10:36.903526+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:10:36.903526+0000","last_undegraded":"2024-01-11T10:10:36.903526+0000","last_fullsized":"2024-01-11T10:10:36.903526+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1540,"log_dups_size":0,"ondisk_log_size":1540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45678592,"num_objects":697,"num_object_clones":0,"num_object_copies":2091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1541,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'1586","reported_seq":1602,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:37.849772+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:10:37.849772+0000","last_peered":"2024-01-11T10:10:37.849772+0000","last_clean":"2024-01-11T10:10:37.849772+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:10:37.849772+0000","last_undegraded":"2024-01-11T10:10:37.849772+0000","last_fullsized":"2024-01-11T10:10:37.849772+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1586,"log_dups_size":0,"ondisk_log_size":1586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49479680,"num_objects":755,"num_object_clones":0,"num_object_copies":2265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1587,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204927+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:10:02.204927+0000","last_peered":"2024-01-11T10:10:02.204927+0000","last_clean":"2024-01-11T10:10:02.204927+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:10:02.204927+0000","last_undegraded":"2024-01-11T10:10:02.204927+0000","last_fullsized":"2024-01-11T10:10:02.204927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'1593","reported_seq":1613,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:36.856577+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:10:36.856577+0000","last_peered":"2024-01-11T10:10:36.856577+0000","last_clean":"2024-01-11T10:10:36.856577+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:10:36.856577+0000","last_undegraded":"2024-01-11T10:10:36.856577+0000","last_fullsized":"2024-01-11T10:10:36.856577+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1593,"log_dups_size":0,"ondisk_log_size":1593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51118080,"num_objects":780,"num_object_clones":0,"num_object_copies":2340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1594,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'1524","reported_seq":1540,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:38.063964+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:10:38.063964+0000","last_peered":"2024-01-11T10:10:38.063964+0000","last_clean":"2024-01-11T10:10:38.063964+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:10:38.063964+0000","last_undegraded":"2024-01-11T10:10:38.063964+0000","last_fullsized":"2024-01-11T10:10:38.063964+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154304,"num_objects":689,"num_object_clones":0,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1525,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.196049+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:10:02.196049+0000","last_peered":"2024-01-11T10:10:02.196049+0000","last_clean":"2024-01-11T10:10:02.196049+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:10:02.196049+0000","last_undegraded":"2024-01-11T10:10:02.196049+0000","last_fullsized":"2024-01-11T10:10:02.196049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":774242328,"num_objects":11815,"num_object_clones":0,"num_object_copies":35445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25127,"num_write_kb":1182081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25111,"ondisk_log_size":25111,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81852,"kb_used_data":32816,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94289988,"statfs":{"total":96636764160,"available":96552947712,"internally_reserved":0,"allocated":33603584,"data_stored":534500809,"data_compressed":25380319,"data_compressed_allocated":33406976,"data_compressed_original":534429696,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705804,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82236,"kb_used_data":33204,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94289604,"statfs":{"total":96636764160,"available":96552554496,"internally_reserved":0,"allocated":34000896,"data_stored":540792289,"data_compressed":25679070,"data_compressed_allocated":33800192,"data_compressed_original":540721152,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91500,"kb_used_data":42468,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94280340,"statfs":{"total":96636764160,"available":96543068160,"internally_reserved":0,"allocated":43487232,"data_stored":692573665,"data_compressed":32886462,"data_compressed_allocated":43286528,"data_compressed_original":692502528,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705803,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":89488,"kb_used_data":36296,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94282352,"statfs":{"total":96636764160,"available":96545128448,"internally_reserved":0,"allocated":37167104,"data_stored":591844289,"data_compressed":28101359,"data_compressed_allocated":36986880,"data_compressed_original":591790080,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:10:41.269 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:41.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:41.414+0000 7f087b75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:41.403 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:41.419+0000 7f087b75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:43.642 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:43.779 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:43.793+0000 7f0d3bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:43.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:43.799+0000 7f0d3bd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:45.423 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:45.423 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:45.423 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:45.423 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:45.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:45.551+0000 7fed8546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:45.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:45.560+0000 7fed8546d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:45.637 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:45.638 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:45.641 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:45.641 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:45.789 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:45.800+0000 7f53ee51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:45.793 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:45.808+0000 7f53ee51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:45.850 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:45.850 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:45.851 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:45.854 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:45.854 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:45.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:45.981+0000 7f877002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:45.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:45.992+0000 7f877002d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:46.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:46.036 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:46.036 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:46.037 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:46.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:46.039 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:46.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:46.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:46.153+0000 7f7d6a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:46.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:46.158+0000 7f7d6a46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:46.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:46.250+0000 7fe9b04f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:46.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:46.258+0000 7fe9b04f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:46.313 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:46.314 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:46.316 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:48.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:48.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:48.589+0000 7fee4c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:48.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:48.594+0000 7fee4c6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:50.855 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:50.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:50.981+0000 7f709b15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:50.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:50.986+0000 7f709b15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.317 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:51.317 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:51.317 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:51.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:51.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.447+0000 7f3d7b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.455+0000 7f3d7b79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:51.499 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:51.500 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:51.500 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:51.500 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:51.500 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:51.500 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:51.500 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:51.502 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:51.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:51.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.630+0000 7f3e7c2de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.641+0000 7f3e7c2de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:51.654 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:51.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:51.657 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:51.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:51.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.786+0000 7f7b8815e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.794+0000 7f7b8815e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.808 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:51.809 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:51.811 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:51.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:51.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.940+0000 7f239519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:51.933 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:51.949+0000 7f239519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:52.013 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:52.014 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:52.016 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:53.180 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:53.326 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:53.340+0000 7f615119e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:53.329 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:53.345+0000 7f615119e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:55.607 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:10:55.721 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:55.735+0000 7f2218206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:55.725 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:55.741+0000 7f2218206640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:10:57.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:10:57.017 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:10:57.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:10:57.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.142+0000 7fbd9e55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.150+0000 7fbd9e55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:57.148 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:57.149 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:57.151 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:57.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:10:57.268 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.277+0000 7febce39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.272 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.288+0000 7febce39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:57.308 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:57.309 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:57.312 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:57.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:10:57.431 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.442+0000 7f2ec7817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.451+0000 7f2ec7817640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:57.453 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:57.454 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:57.456 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:57.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:10:57.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.616+0000 7fe3ec22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:57.626+0000 7fe3ec22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:10:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:10:57.685 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:10:57.686 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:10:57.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:10:57.689 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:10:57.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:10:58.120 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:58.134+0000 7fba04b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:10:58.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:10:58.141+0000 7fba04b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:00.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:00.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:00.506+0000 7fdc1ca60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:00.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:00.512+0000 7fdc1ca60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:02.689 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:02.690 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:02.690 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:11:02.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:11:02.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:02.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:02.838+0000 7f9efb99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:02.831 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:02.847+0000 7f9efb99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:02.897 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:02.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:02.899 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:02.900 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:11:02.916 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:02.930+0000 7ff520cf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:02.920 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:02.936+0000 7ff520cf7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.020 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:03.031+0000 7fe2f362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.025 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:03.041+0000 7fe2f362d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:03.085 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:03.086 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:03.089 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:03.089 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:11:03.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:03.261+0000 7f1262a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.257 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:03.273+0000 7f1262a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.285 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:03.285 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:03.286 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:03.288 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:03.288 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:03.288 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:11:03.432 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:03.443+0000 7f634ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.436 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:03.452+0000 7f634ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:03.451 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:03.452 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:03.452 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:03.452 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:03.455 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:05.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:05.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:05.218+0000 7f89e906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:05.207 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:05.223+0000 7f89e906d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:07.431 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:07.543 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:07.557+0000 7f8bbeb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:07.546 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:07.562+0000 7f8bbeb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:08.455 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:08.456 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:08.456 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:11:08.456 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:11:08.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:08.608+0000 7f0731f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:08.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:08.617+0000 7f0731f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:08.633 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:08.634 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:08.637 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:08.637 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:11:08.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:08.798+0000 7fbd1f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:08.792 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:08.808+0000 7fbd1f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:08.832 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:08.836 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:08.836 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:11:08.961 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:08.972+0000 7f912e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:08.965 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:08.981+0000 7f912e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:09.031 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:09.032 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:09.035 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:09.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:11:09.176 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:09.186+0000 7fcd8a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:09.179 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:09.195+0000 7fcd8a26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:09.279 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:09.282 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:09.773 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:09.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:09.906+0000 7f3668860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:09.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:09.911+0000 7f3668860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:12.172 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:12.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:12.299+0000 7f9af0f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:12.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:12.304+0000 7f9af0f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.282 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:14.283 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:14.283 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:11:14.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:11:14.423 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.435+0000 7fb11c35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.427 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.444+0000 7fb11c35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.446 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:14.447 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:14.450 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:14.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:11:14.549 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:14.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.609+0000 7f6e2ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.605 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.622+0000 7f6e2ea6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.686+0000 7f94cab1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.691+0000 7f94cab1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:14.708 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:14.709 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:14.710 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:14.710 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:14.711 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:11:14.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.859+0000 7fc98ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:14.868+0000 7fc98ad9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:14.884 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:14.885 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:14.887 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:14.887 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:11:15.028 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:15.040+0000 7fc0490f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:15.032 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:15.049+0000 7fc0490f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:15.097 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:15.098 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:15.101 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:16.921 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:17.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:17.051+0000 7fd5a2f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:17.039 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:17.056+0000 7fd5a2f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:19.312 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:19.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:19.473+0000 7f03e399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:19.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:19.478+0000 7f03e399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:20.102 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:20.102 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:20.102 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:20.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:20.234+0000 7f53c935e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:20.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:20.243+0000 7f53c935e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:20.536 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:20.537 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:20.550 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-01-11T10:11:20.128772+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":257884728,"num_objects":3929,"num_object_clones":0,"num_object_copies":11787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":33152,"num_write_kb":1183975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33096,"ondisk_log_size":33096,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":280720,"kb_used_data":49900,"kb_used_omap":0,"kb_used_meta":230784,"kb_avail":377206640,"statfs":{"total":386547056640,"available":386259599360,"internally_reserved":0,"allocated":51097600,"data_stored":805131596,"data_compressed":38227461,"data_compressed_allocated":50319360,"data_compressed_original":804864000,"omap_allocated":0,"internal_metadata":236322816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-154992640,"num_objects":-2365,"num_object_clones":0,"num_object_copies":-7095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2365,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002125"},"pg_stats":[{"pgid":"3.f","version":"21'2025","reported_seq":2041,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:18.192070+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:11:18.192070+0000","last_peered":"2024-01-11T10:11:18.192070+0000","last_clean":"2024-01-11T10:11:18.192070+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:11:18.192070+0000","last_undegraded":"2024-01-11T10:11:18.192070+0000","last_fullsized":"2024-01-11T10:11:18.192070+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'2103","reported_seq":2119,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:18.098803+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:11:18.098803+0000","last_peered":"2024-01-11T10:11:18.098803+0000","last_clean":"2024-01-11T10:11:18.098803+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:11:18.098803+0000","last_undegraded":"2024-01-11T10:11:18.098803+0000","last_fullsized":"2024-01-11T10:11:18.098803+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2103,"log_dups_size":0,"ondisk_log_size":2103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384000,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2104,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'2006","reported_seq":2025,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.051160+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:11:17.051160+0000","last_peered":"2024-01-11T10:11:17.051160+0000","last_clean":"2024-01-11T10:11:17.051160+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:11:17.051160+0000","last_undegraded":"2024-01-11T10:11:17.051160+0000","last_fullsized":"2024-01-11T10:11:17.051160+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16121880,"num_objects":247,"num_object_clones":0,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2007,"num_write_kb":72065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'2043","reported_seq":2059,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:18.120127+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:11:18.120127+0000","last_peered":"2024-01-11T10:11:18.120127+0000","last_clean":"2024-01-11T10:11:18.120127+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:11:18.120127+0000","last_undegraded":"2024-01-11T10:11:18.120127+0000","last_fullsized":"2024-01-11T10:11:18.120127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2043,"log_dups_size":0,"ondisk_log_size":2043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15728640,"num_objects":240,"num_object_clones":0,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2044,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'1992","reported_seq":2012,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:16.986345+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:11:16.986345+0000","last_peered":"2024-01-11T10:11:16.986345+0000","last_clean":"2024-01-11T10:11:16.986345+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:11:16.986345+0000","last_undegraded":"2024-01-11T10:11:16.986345+0000","last_fullsized":"2024-01-11T10:11:16.986345+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1992,"log_dups_size":0,"ondisk_log_size":1992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1993,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'2087","reported_seq":2103,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:18.080439+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:11:18.080439+0000","last_peered":"2024-01-11T10:11:18.080439+0000","last_clean":"2024-01-11T10:11:18.080439+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:11:18.080439+0000","last_undegraded":"2024-01-11T10:11:18.080439+0000","last_fullsized":"2024-01-11T10:11:18.080439+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2087,"log_dups_size":0,"ondisk_log_size":2087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15073280,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'2067","reported_seq":2083,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.928935+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:11:17.928935+0000","last_peered":"2024-01-11T10:11:17.928935+0000","last_clean":"2024-01-11T10:11:17.928935+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:11:17.928935+0000","last_undegraded":"2024-01-11T10:11:17.928935+0000","last_fullsized":"2024-01-11T10:11:17.928935+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2067,"log_dups_size":0,"ondisk_log_size":2067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16908288,"num_objects":258,"num_object_clones":0,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2068,"num_write_kb":74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'2038","reported_seq":2056,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.024573+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:11:17.024573+0000","last_peered":"2024-01-11T10:11:17.024573+0000","last_clean":"2024-01-11T10:11:17.024573+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:11:17.024573+0000","last_undegraded":"2024-01-11T10:11:17.024573+0000","last_fullsized":"2024-01-11T10:11:17.024573+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2038,"log_dups_size":0,"ondisk_log_size":2038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":156,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.205924+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:10:02.205924+0000","last_peered":"2024-01-11T10:10:02.205924+0000","last_clean":"2024-01-11T10:10:02.205924+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:10:02.205924+0000","last_undegraded":"2024-01-11T10:10:02.205924+0000","last_fullsized":"2024-01-11T10:10:02.205924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.182609+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:10:02.182609+0000","last_peered":"2024-01-11T10:10:02.182609+0000","last_clean":"2024-01-11T10:10:02.182609+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:10:02.182609+0000","last_undegraded":"2024-01-11T10:10:02.182609+0000","last_fullsized":"2024-01-11T10:10:02.182609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'2109","reported_seq":2127,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.121443+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:11:17.121443+0000","last_peered":"2024-01-11T10:11:17.121443+0000","last_clean":"2024-01-11T10:11:17.121443+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:11:17.121443+0000","last_undegraded":"2024-01-11T10:11:17.121443+0000","last_fullsized":"2024-01-11T10:11:17.121443+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2110,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'2153","reported_seq":2173,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.096820+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:11:17.096820+0000","last_peered":"2024-01-11T10:11:17.096820+0000","last_clean":"2024-01-11T10:11:17.096820+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:11:17.096820+0000","last_undegraded":"2024-01-11T10:11:17.096820+0000","last_fullsized":"2024-01-11T10:11:17.096820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2153,"log_dups_size":0,"ondisk_log_size":2153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2154,"num_write_kb":76480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.179920+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:10:02.179920+0000","last_peered":"2024-01-11T10:10:02.179920+0000","last_clean":"2024-01-11T10:10:02.179920+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:10:02.179920+0000","last_undegraded":"2024-01-11T10:10:02.179920+0000","last_fullsized":"2024-01-11T10:10:02.179920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'1977","reported_seq":1997,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:16.917566+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:11:16.917566+0000","last_peered":"2024-01-11T10:11:16.917566+0000","last_clean":"2024-01-11T10:11:16.917566+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:11:16.917566+0000","last_undegraded":"2024-01-11T10:11:16.917566+0000","last_fullsized":"2024-01-11T10:11:16.917566+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1977,"log_dups_size":0,"ondisk_log_size":1977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16384000,"num_objects":250,"num_object_clones":0,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1978,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'2218","reported_seq":2234,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.946149+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:11:17.946149+0000","last_peered":"2024-01-11T10:11:17.946149+0000","last_clean":"2024-01-11T10:11:17.946149+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:11:17.946149+0000","last_undegraded":"2024-01-11T10:11:17.946149+0000","last_fullsized":"2024-01-11T10:11:17.946149+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2218,"log_dups_size":0,"ondisk_log_size":2218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16449536,"num_objects":251,"num_object_clones":0,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2219,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'2001","reported_seq":2019,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:17.017540+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:11:17.017540+0000","last_peered":"2024-01-11T10:11:17.017540+0000","last_clean":"2024-01-11T10:11:17.017540+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:11:17.017540+0000","last_undegraded":"2024-01-11T10:11:17.017540+0000","last_fullsized":"2024-01-11T10:11:17.017540+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2001,"log_dups_size":0,"ondisk_log_size":2001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2002,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'2105","reported_seq":2121,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:18.037133+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:11:18.037133+0000","last_peered":"2024-01-11T10:11:18.037133+0000","last_clean":"2024-01-11T10:11:18.037133+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:11:18.037133+0000","last_undegraded":"2024-01-11T10:11:18.037133+0000","last_fullsized":"2024-01-11T10:11:18.037133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2105,"log_dups_size":0,"ondisk_log_size":2105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2106,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204927+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:10:02.204927+0000","last_peered":"2024-01-11T10:10:02.204927+0000","last_clean":"2024-01-11T10:10:02.204927+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:10:02.204927+0000","last_undegraded":"2024-01-11T10:10:02.204927+0000","last_fullsized":"2024-01-11T10:10:02.204927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'2087","reported_seq":2107,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:16.912710+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:11:16.912710+0000","last_peered":"2024-01-11T10:11:16.912710+0000","last_clean":"2024-01-11T10:11:16.912710+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:11:16.912710+0000","last_undegraded":"2024-01-11T10:11:16.912710+0000","last_fullsized":"2024-01-11T10:11:16.912710+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2087,"log_dups_size":0,"ondisk_log_size":2087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18743296,"num_objects":286,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'1988","reported_seq":2004,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:18.040914+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:11:18.040914+0000","last_peered":"2024-01-11T10:11:18.040914+0000","last_clean":"2024-01-11T10:11:18.040914+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:11:18.040914+0000","last_undegraded":"2024-01-11T10:11:18.040914+0000","last_fullsized":"2024-01-11T10:11:18.040914+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":1988,"log_dups_size":0,"ondisk_log_size":1988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745600,"num_objects":225,"num_object_clones":0,"num_object_copies":675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1989,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.196049+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:10:02.196049+0000","last_peered":"2024-01-11T10:10:02.196049+0000","last_clean":"2024-01-11T10:10:02.196049+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:10:02.196049+0000","last_undegraded":"2024-01-11T10:10:02.196049+0000","last_fullsized":"2024-01-11T10:10:02.196049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":257294360,"num_objects":3927,"num_object_clones":0,"num_object_copies":11781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":33015,"num_write_kb":1182081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32999,"ondisk_log_size":32999,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64320,"kb_used_data":11128,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94307520,"statfs":{"total":96636764160,"available":96570900480,"internally_reserved":0,"allocated":11395072,"data_stored":179164617,"data_compressed":8507055,"data_compressed_allocated":11198464,"data_compressed_original":179093504,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705844,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":64424,"kb_used_data":11232,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94307416,"statfs":{"total":96636764160,"available":96570793984,"internally_reserved":0,"allocated":11501568,"data_stored":180803041,"data_compressed":8584855,"data_compressed_allocated":11300864,"data_compressed_original":180731904,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77708,"kb_used_data":14788,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94294132,"statfs":{"total":96636764160,"available":96557191168,"internally_reserved":0,"allocated":15142912,"data_stored":239064545,"data_compressed":11351423,"data_compressed_allocated":14942208,"data_compressed_original":238993408,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705843,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74268,"kb_used_data":12752,"kb_used_omap":0,"kb_used_meta":61504,"kb_avail":94297572,"statfs":{"total":96636764160,"available":96560713728,"internally_reserved":0,"allocated":13058048,"data_stored":206099393,"data_compressed":9784128,"data_compressed_allocated":12877824,"data_compressed_original":206045184,"omap_allocated":0,"internal_metadata":62980096},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:21.720 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:21.867 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:21.882+0000 7f93cac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:21.870 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:21.887+0000 7f93cac6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:24.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:24.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:24.402+0000 7f2d3cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:24.390 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:24.407+0000 7f2d3cb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:25.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:25.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:25.553 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:25.668 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:25.680+0000 7f75cff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:25.672 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:25.689+0000 7f75cff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:25.871 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:25.871 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:25.883 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-01-11T10:11:24.129516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":196215352,"num_objects":2988,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":34093,"num_write_kb":1183975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34037,"ondisk_log_size":34037,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":264976,"kb_used_data":38320,"kb_used_omap":0,"kb_used_meta":226624,"kb_avail":377222384,"statfs":{"total":386547056640,"available":386275721216,"internally_reserved":0,"allocated":39239680,"data_stored":615404876,"data_compressed":29218221,"data_compressed_allocated":38461440,"data_compressed_original":615137280,"omap_allocated":0,"internal_metadata":232062976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-185204736,"num_objects":-2826,"num_object_clones":0,"num_object_copies":-8478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2826,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002209"},"pg_stats":[{"pgid":"3.f","version":"21'2086","reported_seq":2102,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:23.070536+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:11:23.070536+0000","last_peered":"2024-01-11T10:11:23.070536+0000","last_clean":"2024-01-11T10:11:23.070536+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:11:23.070536+0000","last_undegraded":"2024-01-11T10:11:23.070536+0000","last_fullsized":"2024-01-11T10:11:23.070536+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2086,"log_dups_size":0,"ondisk_log_size":2086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11206656,"num_objects":171,"num_object_clones":0,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2087,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'2159","reported_seq":2175,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.673357+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:11:22.673357+0000","last_peered":"2024-01-11T10:11:22.673357+0000","last_clean":"2024-01-11T10:11:22.673357+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:11:22.673357+0000","last_undegraded":"2024-01-11T10:11:22.673357+0000","last_fullsized":"2024-01-11T10:11:22.673357+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2159,"log_dups_size":0,"ondisk_log_size":2159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12713984,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2160,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'2059","reported_seq":2078,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:21.890265+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:11:21.890265+0000","last_peered":"2024-01-11T10:11:21.890265+0000","last_clean":"2024-01-11T10:11:21.890265+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:11:21.890265+0000","last_undegraded":"2024-01-11T10:11:21.890265+0000","last_fullsized":"2024-01-11T10:11:21.890265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2059,"log_dups_size":0,"ondisk_log_size":2059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648472,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2060,"num_write_kb":72065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'2102","reported_seq":2118,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:23.187245+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:11:23.187245+0000","last_peered":"2024-01-11T10:11:23.187245+0000","last_clean":"2024-01-11T10:11:23.187245+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:11:23.187245+0000","last_undegraded":"2024-01-11T10:11:23.187245+0000","last_fullsized":"2024-01-11T10:11:23.187245+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2103,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2042","reported_seq":2062,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:21.980422+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:11:21.980422+0000","last_peered":"2024-01-11T10:11:21.980422+0000","last_clean":"2024-01-11T10:11:21.980422+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:11:21.980422+0000","last_undegraded":"2024-01-11T10:11:21.980422+0000","last_fullsized":"2024-01-11T10:11:21.980422+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2042,"log_dups_size":0,"ondisk_log_size":2042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2043,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'2151","reported_seq":2167,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:23.175415+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:11:23.175415+0000","last_peered":"2024-01-11T10:11:23.175415+0000","last_clean":"2024-01-11T10:11:23.175415+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:11:23.175415+0000","last_undegraded":"2024-01-11T10:11:23.175415+0000","last_fullsized":"2024-01-11T10:11:23.175415+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2151,"log_dups_size":0,"ondisk_log_size":2151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2152,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'2124","reported_seq":2140,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.987481+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:11:22.987481+0000","last_peered":"2024-01-11T10:11:22.987481+0000","last_clean":"2024-01-11T10:11:22.987481+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:11:22.987481+0000","last_undegraded":"2024-01-11T10:11:22.987481+0000","last_fullsized":"2024-01-11T10:11:22.987481+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2124,"log_dups_size":0,"ondisk_log_size":2124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13172736,"num_objects":201,"num_object_clones":0,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'2098","reported_seq":2116,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:21.949294+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:11:21.949294+0000","last_peered":"2024-01-11T10:11:21.949294+0000","last_clean":"2024-01-11T10:11:21.949294+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:11:21.949294+0000","last_undegraded":"2024-01-11T10:11:21.949294+0000","last_fullsized":"2024-01-11T10:11:21.949294+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2099,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":156,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.205924+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:10:02.205924+0000","last_peered":"2024-01-11T10:10:02.205924+0000","last_clean":"2024-01-11T10:10:02.205924+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:10:02.205924+0000","last_undegraded":"2024-01-11T10:10:02.205924+0000","last_fullsized":"2024-01-11T10:10:02.205924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.182609+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:10:02.182609+0000","last_peered":"2024-01-11T10:10:02.182609+0000","last_clean":"2024-01-11T10:10:02.182609+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:10:02.182609+0000","last_undegraded":"2024-01-11T10:10:02.182609+0000","last_fullsized":"2024-01-11T10:10:02.182609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'2166","reported_seq":2184,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.052802+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:11:22.052802+0000","last_peered":"2024-01-11T10:11:22.052802+0000","last_clean":"2024-01-11T10:11:22.052802+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:11:22.052802+0000","last_undegraded":"2024-01-11T10:11:22.052802+0000","last_fullsized":"2024-01-11T10:11:22.052802+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2166,"log_dups_size":0,"ondisk_log_size":2166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14090240,"num_objects":215,"num_object_clones":0,"num_object_copies":645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2167,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'2207","reported_seq":2227,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.153735+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:11:22.153735+0000","last_peered":"2024-01-11T10:11:22.153735+0000","last_clean":"2024-01-11T10:11:22.153735+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:11:22.153735+0000","last_undegraded":"2024-01-11T10:11:22.153735+0000","last_fullsized":"2024-01-11T10:11:22.153735+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":76480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.179920+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:10:02.179920+0000","last_peered":"2024-01-11T10:10:02.179920+0000","last_clean":"2024-01-11T10:10:02.179920+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:10:02.179920+0000","last_undegraded":"2024-01-11T10:10:02.179920+0000","last_fullsized":"2024-01-11T10:10:02.179920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'2036","reported_seq":2056,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:21.814765+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:11:21.814765+0000","last_peered":"2024-01-11T10:11:21.814765+0000","last_clean":"2024-01-11T10:11:21.814765+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:11:21.814765+0000","last_undegraded":"2024-01-11T10:11:21.814765+0000","last_fullsized":"2024-01-11T10:11:21.814765+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2036,"log_dups_size":0,"ondisk_log_size":2036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2037,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'2274","reported_seq":2290,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.977097+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:11:22.977097+0000","last_peered":"2024-01-11T10:11:22.977097+0000","last_clean":"2024-01-11T10:11:22.977097+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:11:22.977097+0000","last_undegraded":"2024-01-11T10:11:22.977097+0000","last_fullsized":"2024-01-11T10:11:22.977097+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2275,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'2057","reported_seq":2075,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.041919+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:11:22.041919+0000","last_peered":"2024-01-11T10:11:22.041919+0000","last_clean":"2024-01-11T10:11:22.041919+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:11:22.041919+0000","last_undegraded":"2024-01-11T10:11:22.041919+0000","last_fullsized":"2024-01-11T10:11:22.041919+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2057,"log_dups_size":0,"ondisk_log_size":2057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11796480,"num_objects":180,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2058,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'2160","reported_seq":2176,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:23.139081+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:11:23.139081+0000","last_peered":"2024-01-11T10:11:23.139081+0000","last_clean":"2024-01-11T10:11:23.139081+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:11:23.139081+0000","last_undegraded":"2024-01-11T10:11:23.139081+0000","last_fullsized":"2024-01-11T10:11:23.139081+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2160,"log_dups_size":0,"ondisk_log_size":2160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11862016,"num_objects":181,"num_object_clones":0,"num_object_copies":543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2161,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204927+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:10:02.204927+0000","last_peered":"2024-01-11T10:10:02.204927+0000","last_clean":"2024-01-11T10:10:02.204927+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:10:02.204927+0000","last_undegraded":"2024-01-11T10:10:02.204927+0000","last_fullsized":"2024-01-11T10:10:02.204927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'2167","reported_seq":2187,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:22.033760+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:11:22.033760+0000","last_peered":"2024-01-11T10:11:22.033760+0000","last_clean":"2024-01-11T10:11:22.033760+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:11:22.033760+0000","last_undegraded":"2024-01-11T10:11:22.033760+0000","last_fullsized":"2024-01-11T10:11:22.033760+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13500416,"num_objects":206,"num_object_clones":0,"num_object_copies":618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2168,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'2052","reported_seq":2068,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:23.095921+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:11:23.095921+0000","last_peered":"2024-01-11T10:11:23.095921+0000","last_clean":"2024-01-11T10:11:23.095921+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:11:23.095921+0000","last_undegraded":"2024-01-11T10:11:23.095921+0000","last_fullsized":"2024-01-11T10:11:23.095921+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2053,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.196049+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:10:02.196049+0000","last_peered":"2024-01-11T10:10:02.196049+0000","last_clean":"2024-01-11T10:10:02.196049+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:10:02.196049+0000","last_undegraded":"2024-01-11T10:10:02.196049+0000","last_fullsized":"2024-01-11T10:10:02.196049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":195624984,"num_objects":2986,"num_object_clones":0,"num_object_copies":8958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":33956,"num_write_kb":1182081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33940,"ondisk_log_size":33940,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65832,"kb_used_data":8480,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94306008,"statfs":{"total":96636764160,"available":96569352192,"internally_reserved":0,"allocated":8683520,"data_stored":135779785,"data_compressed":6446911,"data_compressed_allocated":8486912,"data_compressed_original":135708672,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705849,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":66008,"kb_used_data":8656,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":94305832,"statfs":{"total":96636764160,"available":96569171968,"internally_reserved":0,"allocated":8863744,"data_stored":138597857,"data_compressed":6580727,"data_compressed_allocated":8663040,"data_compressed_original":138526720,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":78488,"kb_used_data":11408,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94293352,"statfs":{"total":96636764160,"available":96556392448,"internally_reserved":0,"allocated":11681792,"data_stored":183686625,"data_compressed":8721783,"data_compressed_allocated":11481088,"data_compressed_original":183615488,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705848,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54648,"kb_used_data":9776,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94317192,"statfs":{"total":96636764160,"available":96580804608,"internally_reserved":0,"allocated":10010624,"data_stored":157340609,"data_compressed":7468800,"data_compressed_allocated":9830400,"data_compressed_original":157286400,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:26.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:26.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:26.942+0000 7fe03526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:26.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:26.946+0000 7fe03526d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:29.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:29.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:29.303+0000 7ffa2239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:29.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:29.308+0000 7ffa2239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:30.885 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:30.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:30.885 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:11:30.886 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:11:31.001 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.013+0000 7ff1914de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.022+0000 7ff1914de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:31.070 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:31.074 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:31.074 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:11:31.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.227+0000 7fc61caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.220 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.237+0000 7fc61caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.264 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:31.264 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:31.264 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:31.264 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:31.264 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:31.265 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:31.268 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:31.268 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:11:31.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.395+0000 7ffaa471e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.404+0000 7ffaa471e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:31.403 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:31.404 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:31.406 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:31.407 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:11:31.526 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.535+0000 7f922599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.529 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.546+0000 7f922599e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:31.604 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:31.605 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:31.605 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:31.605 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:31.607 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:31.659 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:31.771 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.786+0000 7f327682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:31.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:31.791+0000 7f327682d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:34.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:34.267 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:34.282+0000 7fa87e31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:34.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:34.287+0000 7fa87e31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:36.607 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:36.607 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:36.608 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:36.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:36.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:36.752+0000 7ff3bdf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:36.744 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:36.761+0000 7ff3bdf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:36.787 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:36.803+0000 7fb83991e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:36.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:36.808+0000 7fb83991e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:37.063 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:37.064 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:37.077 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-01-11T10:11:36.131760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":73400888,"num_objects":1114,"num_object_clones":0,"num_object_copies":3342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":35967,"num_write_kb":1183975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35911,"ondisk_log_size":35911,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":230000,"kb_used_data":15820,"kb_used_omap":0,"kb_used_meta":214144,"kb_avail":377257360,"statfs":{"total":386547056640,"available":386311536640,"internally_reserved":0,"allocated":16199680,"data_stored":246764876,"data_compressed":11713221,"data_compressed_allocated":15421440,"data_compressed_original":246497280,"omap_allocated":0,"internal_metadata":219283456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-122814464,"num_objects":-1874,"num_object_clones":0,"num_object_copies":-5622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1874,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002243"},"pg_stats":[{"pgid":"3.f","version":"21'2195","reported_seq":2211,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:33.162029+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:11:33.162029+0000","last_peered":"2024-01-11T10:11:33.162029+0000","last_clean":"2024-01-11T10:11:33.162029+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:11:33.162029+0000","last_undegraded":"2024-01-11T10:11:33.162029+0000","last_fullsized":"2024-01-11T10:11:33.162029+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2195,"log_dups_size":0,"ondisk_log_size":2195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4063232,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'2274","reported_seq":2290,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:33.028045+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:11:33.028045+0000","last_peered":"2024-01-11T10:11:33.028045+0000","last_clean":"2024-01-11T10:11:33.028045+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:11:33.028045+0000","last_undegraded":"2024-01-11T10:11:33.028045+0000","last_fullsized":"2024-01-11T10:11:33.028045+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2274,"log_dups_size":0,"ondisk_log_size":2274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5177344,"num_objects":79,"num_object_clones":0,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2275,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'2179","reported_seq":2198,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:31.968744+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:11:31.968744+0000","last_peered":"2024-01-11T10:11:31.968744+0000","last_clean":"2024-01-11T10:11:31.968744+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:11:31.968744+0000","last_undegraded":"2024-01-11T10:11:31.968744+0000","last_fullsized":"2024-01-11T10:11:31.968744+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784152,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2180,"num_write_kb":72065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'2227","reported_seq":2243,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.947516+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:11:32.947516+0000","last_peered":"2024-01-11T10:11:32.947516+0000","last_clean":"2024-01-11T10:11:32.947516+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:11:32.947516+0000","last_undegraded":"2024-01-11T10:11:32.947516+0000","last_fullsized":"2024-01-11T10:11:32.947516+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2227,"log_dups_size":0,"ondisk_log_size":2227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3670016,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2228,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2142","reported_seq":2162,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.075428+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:11:32.075428+0000","last_peered":"2024-01-11T10:11:32.075428+0000","last_clean":"2024-01-11T10:11:32.075428+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:11:32.075428+0000","last_undegraded":"2024-01-11T10:11:32.075428+0000","last_fullsized":"2024-01-11T10:11:32.075428+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2142,"log_dups_size":0,"ondisk_log_size":2142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2143,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'2260","reported_seq":2276,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:33.063135+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:11:33.063135+0000","last_peered":"2024-01-11T10:11:33.063135+0000","last_clean":"2024-01-11T10:11:33.063135+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:11:33.063135+0000","last_undegraded":"2024-01-11T10:11:33.063135+0000","last_fullsized":"2024-01-11T10:11:33.063135+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2260,"log_dups_size":0,"ondisk_log_size":2260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2261,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'2249","reported_seq":2265,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:33.126001+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:11:33.126001+0000","last_peered":"2024-01-11T10:11:33.126001+0000","last_clean":"2024-01-11T10:11:33.126001+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:11:33.126001+0000","last_undegraded":"2024-01-11T10:11:33.126001+0000","last_fullsized":"2024-01-11T10:11:33.126001+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4980736,"num_objects":76,"num_object_clones":0,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'2213","reported_seq":2231,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:31.923055+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:11:31.923055+0000","last_peered":"2024-01-11T10:11:31.923055+0000","last_clean":"2024-01-11T10:11:31.923055+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:11:31.923055+0000","last_undegraded":"2024-01-11T10:11:31.923055+0000","last_fullsized":"2024-01-11T10:11:31.923055+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2213,"log_dups_size":0,"ondisk_log_size":2213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5242880,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2214,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":156,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.205924+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:10:02.205924+0000","last_peered":"2024-01-11T10:10:02.205924+0000","last_clean":"2024-01-11T10:10:02.205924+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:10:02.205924+0000","last_undegraded":"2024-01-11T10:10:02.205924+0000","last_fullsized":"2024-01-11T10:10:02.205924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.182609+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:10:02.182609+0000","last_peered":"2024-01-11T10:10:02.182609+0000","last_clean":"2024-01-11T10:10:02.182609+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:10:02.182609+0000","last_undegraded":"2024-01-11T10:10:02.182609+0000","last_fullsized":"2024-01-11T10:10:02.182609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'2310","reported_seq":2328,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.071310+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:11:32.071310+0000","last_peered":"2024-01-11T10:11:32.071310+0000","last_clean":"2024-01-11T10:11:32.071310+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:11:32.071310+0000","last_undegraded":"2024-01-11T10:11:32.071310+0000","last_fullsized":"2024-01-11T10:11:32.071310+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2310,"log_dups_size":0,"ondisk_log_size":2310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4653056,"num_objects":71,"num_object_clones":0,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2311,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'2320","reported_seq":2340,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.143034+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:11:32.143034+0000","last_peered":"2024-01-11T10:11:32.143034+0000","last_clean":"2024-01-11T10:11:32.143034+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:11:32.143034+0000","last_undegraded":"2024-01-11T10:11:32.143034+0000","last_fullsized":"2024-01-11T10:11:32.143034+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2320,"log_dups_size":0,"ondisk_log_size":2320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4521984,"num_objects":69,"num_object_clones":0,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2321,"num_write_kb":76480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.179920+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:10:02.179920+0000","last_peered":"2024-01-11T10:10:02.179920+0000","last_clean":"2024-01-11T10:10:02.179920+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:10:02.179920+0000","last_undegraded":"2024-01-11T10:10:02.179920+0000","last_fullsized":"2024-01-11T10:10:02.179920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'2154","reported_seq":2174,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.112156+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:11:32.112156+0000","last_peered":"2024-01-11T10:11:32.112156+0000","last_clean":"2024-01-11T10:11:32.112156+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:11:32.112156+0000","last_undegraded":"2024-01-11T10:11:32.112156+0000","last_fullsized":"2024-01-11T10:11:32.112156+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2154,"log_dups_size":0,"ondisk_log_size":2154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'2406","reported_seq":2422,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.986904+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:11:32.986904+0000","last_peered":"2024-01-11T10:11:32.986904+0000","last_clean":"2024-01-11T10:11:32.986904+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:11:32.986904+0000","last_undegraded":"2024-01-11T10:11:32.986904+0000","last_fullsized":"2024-01-11T10:11:32.986904+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4128768,"num_objects":63,"num_object_clones":0,"num_object_copies":189,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'2165","reported_seq":2183,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:31.799962+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:11:31.799962+0000","last_peered":"2024-01-11T10:11:31.799962+0000","last_clean":"2024-01-11T10:11:31.799962+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:11:31.799962+0000","last_undegraded":"2024-01-11T10:11:31.799962+0000","last_fullsized":"2024-01-11T10:11:31.799962+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2165,"log_dups_size":0,"ondisk_log_size":2165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4718592,"num_objects":72,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2166,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'2283","reported_seq":2299,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.910748+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:11:32.910748+0000","last_peered":"2024-01-11T10:11:32.910748+0000","last_clean":"2024-01-11T10:11:32.910748+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:11:32.910748+0000","last_undegraded":"2024-01-11T10:11:32.910748+0000","last_fullsized":"2024-01-11T10:11:32.910748+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2283,"log_dups_size":0,"ondisk_log_size":2283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3801088,"num_objects":58,"num_object_clones":0,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2284,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204927+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:10:02.204927+0000","last_peered":"2024-01-11T10:10:02.204927+0000","last_clean":"2024-01-11T10:10:02.204927+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:10:02.204927+0000","last_undegraded":"2024-01-11T10:10:02.204927+0000","last_fullsized":"2024-01-11T10:10:02.204927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'2276","reported_seq":2296,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.094478+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:11:32.094478+0000","last_peered":"2024-01-11T10:11:32.094478+0000","last_clean":"2024-01-11T10:11:32.094478+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:11:32.094478+0000","last_undegraded":"2024-01-11T10:11:32.094478+0000","last_fullsized":"2024-01-11T10:11:32.094478+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2276,"log_dups_size":0,"ondisk_log_size":2276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'2161","reported_seq":2177,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:32.809655+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:11:32.809655+0000","last_peered":"2024-01-11T10:11:32.809655+0000","last_clean":"2024-01-11T10:11:32.809655+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:11:32.809655+0000","last_undegraded":"2024-01-11T10:11:32.809655+0000","last_fullsized":"2024-01-11T10:11:32.809655+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2161,"log_dups_size":0,"ondisk_log_size":2161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2162,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.196049+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:10:02.196049+0000","last_peered":"2024-01-11T10:10:02.196049+0000","last_clean":"2024-01-11T10:10:02.196049+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:10:02.196049+0000","last_undegraded":"2024-01-11T10:10:02.196049+0000","last_fullsized":"2024-01-11T10:10:02.196049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":72810520,"num_objects":1112,"num_object_clones":0,"num_object_copies":3336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":35830,"num_write_kb":1182081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35814,"ondisk_log_size":35814,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48236,"kb_used_data":3364,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94323604,"statfs":{"total":96636764160,"available":96587370496,"internally_reserved":0,"allocated":3444736,"data_stored":51959241,"data_compressed":2466663,"data_compressed_allocated":3248128,"data_compressed_original":51888128,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705859,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48264,"kb_used_data":3392,"kb_used_omap":0,"kb_used_meta":44864,"kb_avail":94323576,"statfs":{"total":96636764160,"available":96587341824,"internally_reserved":0,"allocated":3473408,"data_stored":52352481,"data_compressed":2485335,"data_compressed_allocated":3272704,"data_compressed_original":52281344,"omap_allocated":0,"internal_metadata":45940736},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76080,"kb_used_data":4840,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94295760,"statfs":{"total":96636764160,"available":96558858240,"internally_reserved":0,"allocated":4956160,"data_stored":76076513,"data_compressed":3611879,"data_compressed_allocated":4755456,"data_compressed_original":76005376,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705858,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":57420,"kb_used_data":4224,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94314420,"statfs":{"total":96636764160,"available":96577966080,"internally_reserved":0,"allocated":4325376,"data_stored":66376641,"data_compressed":3149344,"data_compressed_allocated":4145152,"data_compressed_original":66322432,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:38.763 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Removed 18470 objects 2024-01-11T10:11:38.763 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Clean up completed and total clean up time :92.7103 2024-01-11T10:11:39.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:39.125 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:39.140+0000 7f4bc951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:39.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:39.144+0000 7f4bc951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:41.200 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:41.317 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:41.333+0000 7f99d1708640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:41.320 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:41.337+0000 7f99d1708640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:42.079 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:42.079 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:42.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:42.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:42.213+0000 7f688e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:42.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:42.220+0000 7f688e2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:42.387 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:42.388 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:42.399 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-01-11T10:11:42.132914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":13238840,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":36885,"num_write_kb":1183975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36829,"ondisk_log_size":36829,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":231312,"kb_used_data":4656,"kb_used_omap":0,"kb_used_meta":226624,"kb_avail":377256048,"statfs":{"total":386547056640,"available":386310193152,"internally_reserved":0,"allocated":4767744,"data_stored":63853900,"data_compressed":3027629,"data_compressed_allocated":3989504,"data_compressed_original":63586304,"omap_allocated":0,"internal_metadata":232062976},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-121438208,"num_objects":-1853,"num_object_clones":0,"num_object_copies":-5559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1853,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002203"},"pg_stats":[{"pgid":"3.f","version":"21'2257","reported_seq":2273,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:38.132580+0000","last_change":"2024-01-11T10:08:34.144371+0000","last_active":"2024-01-11T10:11:38.132580+0000","last_peered":"2024-01-11T10:11:38.132580+0000","last_clean":"2024-01-11T10:11:38.132580+0000","last_became_active":"2024-01-11T10:08:34.141450+0000","last_became_peered":"2024-01-11T10:08:34.141450+0000","last_unstale":"2024-01-11T10:11:38.132580+0000","last_undegraded":"2024-01-11T10:11:38.132580+0000","last_fullsized":"2024-01-11T10:11:38.132580+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2257,"log_dups_size":0,"ondisk_log_size":2257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2258,"num_write_kb":72256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"21'2346","reported_seq":2362,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.970832+0000","last_change":"2024-01-11T10:08:34.092804+0000","last_active":"2024-01-11T10:11:37.970832+0000","last_peered":"2024-01-11T10:11:37.970832+0000","last_clean":"2024-01-11T10:11:37.970832+0000","last_became_active":"2024-01-11T10:08:34.089188+0000","last_became_peered":"2024-01-11T10:08:34.089188+0000","last_unstale":"2024-01-11T10:11:37.970832+0000","last_undegraded":"2024-01-11T10:11:37.970832+0000","last_fullsized":"2024-01-11T10:11:37.970832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2346,"log_dups_size":0,"ondisk_log_size":2346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2347,"num_write_kb":75328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"21'2234","reported_seq":2253,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:36.825270+0000","last_change":"2024-01-11T10:08:34.160096+0000","last_active":"2024-01-11T10:11:36.825270+0000","last_peered":"2024-01-11T10:11:36.825270+0000","last_clean":"2024-01-11T10:11:36.825270+0000","last_became_active":"2024-01-11T10:08:34.149932+0000","last_became_peered":"2024-01-11T10:08:34.149932+0000","last_unstale":"2024-01-11T10:11:36.825270+0000","last_undegraded":"2024-01-11T10:11:36.825270+0000","last_fullsized":"2024-01-11T10:11:36.825270+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179672,"num_objects":19,"num_object_clones":0,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2235,"num_write_kb":72065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"21'2277","reported_seq":2293,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.937130+0000","last_change":"2024-01-11T10:08:34.137164+0000","last_active":"2024-01-11T10:11:37.937130+0000","last_peered":"2024-01-11T10:11:37.937130+0000","last_clean":"2024-01-11T10:11:37.937130+0000","last_became_active":"2024-01-11T10:08:34.131286+0000","last_became_peered":"2024-01-11T10:08:34.131286+0000","last_unstale":"2024-01-11T10:11:37.937130+0000","last_undegraded":"2024-01-11T10:11:37.937130+0000","last_fullsized":"2024-01-11T10:11:37.937130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2277,"log_dups_size":0,"ondisk_log_size":2277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2278,"num_write_kb":73088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"21'2199","reported_seq":2219,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:36.655940+0000","last_change":"2024-01-11T10:08:34.156098+0000","last_active":"2024-01-11T10:11:36.655940+0000","last_peered":"2024-01-11T10:11:36.655940+0000","last_clean":"2024-01-11T10:11:36.655940+0000","last_became_active":"2024-01-11T10:08:34.153308+0000","last_became_peered":"2024-01-11T10:08:34.153308+0000","last_unstale":"2024-01-11T10:11:36.655940+0000","last_undegraded":"2024-01-11T10:11:36.655940+0000","last_fullsized":"2024-01-11T10:11:36.655940+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2199,"log_dups_size":0,"ondisk_log_size":2199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2200,"num_write_kb":70912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"21'2308","reported_seq":2324,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:38.000392+0000","last_change":"2024-01-11T10:08:34.130016+0000","last_active":"2024-01-11T10:11:38.000392+0000","last_peered":"2024-01-11T10:11:38.000392+0000","last_clean":"2024-01-11T10:11:38.000392+0000","last_became_active":"2024-01-11T10:08:34.127075+0000","last_became_peered":"2024-01-11T10:08:34.127075+0000","last_unstale":"2024-01-11T10:11:38.000392+0000","last_undegraded":"2024-01-11T10:11:38.000392+0000","last_fullsized":"2024-01-11T10:11:38.000392+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2308,"log_dups_size":0,"ondisk_log_size":2308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":589824,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2309,"num_write_kb":74176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"21'2321","reported_seq":2337,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:38.061469+0000","last_change":"2024-01-11T10:08:34.144827+0000","last_active":"2024-01-11T10:11:38.061469+0000","last_peered":"2024-01-11T10:11:38.061469+0000","last_clean":"2024-01-11T10:11:38.061469+0000","last_became_active":"2024-01-11T10:08:34.141995+0000","last_became_peered":"2024-01-11T10:08:34.141995+0000","last_unstale":"2024-01-11T10:11:38.061469+0000","last_undegraded":"2024-01-11T10:11:38.061469+0000","last_fullsized":"2024-01-11T10:11:38.061469+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2321,"log_dups_size":0,"ondisk_log_size":2321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2322,"num_write_kb":74432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"21'2278","reported_seq":2296,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.148582+0000","last_change":"2024-01-11T10:08:34.166809+0000","last_active":"2024-01-11T10:11:37.148582+0000","last_peered":"2024-01-11T10:11:37.148582+0000","last_clean":"2024-01-11T10:11:37.148582+0000","last_became_active":"2024-01-11T10:08:34.163927+0000","last_became_peered":"2024-01-11T10:08:34.163927+0000","last_unstale":"2024-01-11T10:11:37.148582+0000","last_undegraded":"2024-01-11T10:11:37.148582+0000","last_fullsized":"2024-01-11T10:11:37.148582+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2278,"log_dups_size":0,"ondisk_log_size":2278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":983040,"num_objects":15,"num_object_clones":0,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2279,"num_write_kb":73408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":156,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.205924+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:10:02.205924+0000","last_peered":"2024-01-11T10:10:02.205924+0000","last_clean":"2024-01-11T10:10:02.205924+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:10:02.205924+0000","last_undegraded":"2024-01-11T10:10:02.205924+0000","last_fullsized":"2024-01-11T10:10:02.205924+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.182609+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:10:02.182609+0000","last_peered":"2024-01-11T10:10:02.182609+0000","last_clean":"2024-01-11T10:10:02.182609+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:10:02.182609+0000","last_undegraded":"2024-01-11T10:10:02.182609+0000","last_fullsized":"2024-01-11T10:10:02.182609+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"21'2363","reported_seq":2381,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.063726+0000","last_change":"2024-01-11T10:08:34.124573+0000","last_active":"2024-01-11T10:11:37.063726+0000","last_peered":"2024-01-11T10:11:37.063726+0000","last_clean":"2024-01-11T10:11:37.063726+0000","last_became_active":"2024-01-11T10:08:34.120450+0000","last_became_peered":"2024-01-11T10:08:34.120450+0000","last_unstale":"2024-01-11T10:11:37.063726+0000","last_undegraded":"2024-01-11T10:11:37.063726+0000","last_fullsized":"2024-01-11T10:11:37.063726+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2363,"log_dups_size":0,"ondisk_log_size":2363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1179648,"num_objects":18,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2364,"num_write_kb":76224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.194753+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:10:02.194753+0000","last_peered":"2024-01-11T10:10:02.194753+0000","last_clean":"2024-01-11T10:10:02.194753+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:10:02.194753+0000","last_undegraded":"2024-01-11T10:10:02.194753+0000","last_fullsized":"2024-01-11T10:10:02.194753+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"21'2377","reported_seq":2397,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.117741+0000","last_change":"2024-01-11T10:08:34.183931+0000","last_active":"2024-01-11T10:11:37.117741+0000","last_peered":"2024-01-11T10:11:37.117741+0000","last_clean":"2024-01-11T10:11:37.117741+0000","last_became_active":"2024-01-11T10:08:34.177726+0000","last_became_peered":"2024-01-11T10:08:34.177726+0000","last_unstale":"2024-01-11T10:11:37.117741+0000","last_undegraded":"2024-01-11T10:11:37.117741+0000","last_fullsized":"2024-01-11T10:11:37.117741+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2377,"log_dups_size":0,"ondisk_log_size":2377,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":786432,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2378,"num_write_kb":76480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.179920+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:10:02.179920+0000","last_peered":"2024-01-11T10:10:02.179920+0000","last_clean":"2024-01-11T10:10:02.179920+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:10:02.179920+0000","last_undegraded":"2024-01-11T10:10:02.179920+0000","last_fullsized":"2024-01-11T10:10:02.179920+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"21'2200","reported_seq":2220,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.164074+0000","last_change":"2024-01-11T10:08:34.131417+0000","last_active":"2024-01-11T10:11:37.164074+0000","last_peered":"2024-01-11T10:11:37.164074+0000","last_clean":"2024-01-11T10:11:37.164074+0000","last_became_active":"2024-01-11T10:08:34.128139+0000","last_became_peered":"2024-01-11T10:08:34.128139+0000","last_unstale":"2024-01-11T10:11:37.164074+0000","last_undegraded":"2024-01-11T10:11:37.164074+0000","last_fullsized":"2024-01-11T10:11:37.164074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1769472,"num_objects":27,"num_object_clones":0,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2201,"num_write_kb":71296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204284+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:10:02.204284+0000","last_peered":"2024-01-11T10:10:02.204284+0000","last_clean":"2024-01-11T10:10:02.204284+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:10:02.204284+0000","last_undegraded":"2024-01-11T10:10:02.204284+0000","last_fullsized":"2024-01-11T10:10:02.204284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"21'2463","reported_seq":2479,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:38.130699+0000","last_change":"2024-01-11T10:08:34.108539+0000","last_active":"2024-01-11T10:11:38.130699+0000","last_peered":"2024-01-11T10:11:38.130699+0000","last_clean":"2024-01-11T10:11:38.130699+0000","last_became_active":"2024-01-11T10:08:34.104849+0000","last_became_peered":"2024-01-11T10:08:34.104849+0000","last_unstale":"2024-01-11T10:11:38.130699+0000","last_undegraded":"2024-01-11T10:11:38.130699+0000","last_fullsized":"2024-01-11T10:11:38.130699+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2463,"log_dups_size":0,"ondisk_log_size":2463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2464,"num_write_kb":79040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.5","version":"21'2217","reported_seq":2235,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.097041+0000","last_change":"2024-01-11T10:08:34.180259+0000","last_active":"2024-01-11T10:11:37.097041+0000","last_peered":"2024-01-11T10:11:37.097041+0000","last_clean":"2024-01-11T10:11:37.097041+0000","last_became_active":"2024-01-11T10:08:34.177278+0000","last_became_peered":"2024-01-11T10:08:34.177278+0000","last_unstale":"2024-01-11T10:11:37.097041+0000","last_undegraded":"2024-01-11T10:11:37.097041+0000","last_fullsized":"2024-01-11T10:11:37.097041+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2217,"log_dups_size":0,"ondisk_log_size":2217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1310720,"num_objects":20,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2218,"num_write_kb":71616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.083530+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:10:00.083530+0000","last_peered":"2024-01-11T10:10:00.083530+0000","last_clean":"2024-01-11T10:10:00.083530+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:10:00.083530+0000","last_undegraded":"2024-01-11T10:10:00.083530+0000","last_fullsized":"2024-01-11T10:10:00.083530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"21'2337","reported_seq":2353,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:38.085461+0000","last_change":"2024-01-11T10:08:34.125444+0000","last_active":"2024-01-11T10:11:38.085461+0000","last_peered":"2024-01-11T10:11:38.085461+0000","last_clean":"2024-01-11T10:11:38.085461+0000","last_became_active":"2024-01-11T10:08:34.121153+0000","last_became_peered":"2024-01-11T10:08:34.121153+0000","last_unstale":"2024-01-11T10:11:38.085461+0000","last_undegraded":"2024-01-11T10:11:38.085461+0000","last_fullsized":"2024-01-11T10:11:38.085461+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2337,"log_dups_size":0,"ondisk_log_size":2337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2338,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.204927+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:10:02.204927+0000","last_peered":"2024-01-11T10:10:02.204927+0000","last_clean":"2024-01-11T10:10:02.204927+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:10:02.204927+0000","last_undegraded":"2024-01-11T10:10:02.204927+0000","last_fullsized":"2024-01-11T10:10:02.204927+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'2348","reported_seq":2368,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.095064+0000","last_change":"2024-01-11T10:08:34.151492+0000","last_active":"2024-01-11T10:11:37.095064+0000","last_peered":"2024-01-11T10:11:37.095064+0000","last_clean":"2024-01-11T10:11:37.095064+0000","last_became_active":"2024-01-11T10:08:34.148219+0000","last_became_peered":"2024-01-11T10:08:34.148219+0000","last_unstale":"2024-01-11T10:11:37.095064+0000","last_undegraded":"2024-01-11T10:11:37.095064+0000","last_fullsized":"2024-01-11T10:11:37.095064+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2348,"log_dups_size":0,"ondisk_log_size":2348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1638400,"num_objects":25,"num_object_clones":0,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2349,"num_write_kb":75968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:00.103637+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:10:00.103637+0000","last_peered":"2024-01-11T10:10:00.103637+0000","last_clean":"2024-01-11T10:10:00.103637+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:10:00.103637+0000","last_undegraded":"2024-01-11T10:10:00.103637+0000","last_fullsized":"2024-01-11T10:10:00.103637+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"21'2207","reported_seq":2223,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:11:37.992780+0000","last_change":"2024-01-11T10:08:34.147244+0000","last_active":"2024-01-11T10:11:37.992780+0000","last_peered":"2024-01-11T10:11:37.992780+0000","last_clean":"2024-01-11T10:11:37.992780+0000","last_became_active":"2024-01-11T10:08:34.144047+0000","last_became_peered":"2024-01-11T10:08:34.144047+0000","last_unstale":"2024-01-11T10:11:37.992780+0000","last_undegraded":"2024-01-11T10:11:37.992780+0000","last_fullsized":"2024-01-11T10:11:37.992780+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:32.950699+0000","last_clean_scrub_stamp":"2024-01-11T10:08:32.950699+0000","objects_scrubbed":0,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2208,"num_write_kb":70848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-11T10:10:02.196049+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:10:02.196049+0000","last_peered":"2024-01-11T10:10:02.196049+0000","last_clean":"2024-01-11T10:10:02.196049+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:10:02.196049+0000","last_undegraded":"2024-01-11T10:10:02.196049+0000","last_fullsized":"2024-01-11T10:10:02.196049+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":12648472,"num_objects":194,"num_object_clones":0,"num_object_copies":582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":36748,"num_write_kb":1182081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36732,"ondisk_log_size":36732,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49892,"kb_used_data":860,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94321948,"statfs":{"total":96636764160,"available":96585674752,"internally_reserved":0,"allocated":880640,"data_stored":10933705,"data_compressed":518551,"data_compressed_allocated":684032,"data_compressed_original":10862592,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":49860,"kb_used_data":828,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94321980,"statfs":{"total":96636764160,"available":96585707520,"internally_reserved":0,"allocated":847872,"data_stored":10343905,"data_compressed":490543,"data_compressed_allocated":647168,"data_compressed_original":10272768,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":76964,"kb_used_data":1564,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94294876,"statfs":{"total":96636764160,"available":96557953024,"internally_reserved":0,"allocated":1601536,"data_stored":22402529,"data_compressed":1063151,"data_compressed_allocated":1400832,"data_compressed_original":22331392,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":54596,"kb_used_data":1404,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94317244,"statfs":{"total":96636764160,"available":96580857856,"internally_reserved":0,"allocated":1437696,"data_stored":20173761,"data_compressed":955384,"data_compressed_allocated":1257472,"data_compressed_original":20119552,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:43.396 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:43.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:43.530+0000 7f4dd8b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:43.517 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:43.534+0000 7f4dd8b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:44.104 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-01-11T10:11:44.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-01-11T10:11:44.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:44.233+0000 7f6dd84de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:44.226 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:44.244+0000 7f6dd84de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:44.973 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_0' removed 2024-01-11T10:11:44.989 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-01-11T10:11:44.990 INFO:tasks.radosbench:Beginning radosbench... 2024-01-11T10:11:44.990 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_1 2024-01-11T10:11:44.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_1 16 2024-01-11T10:11:45.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:45.118+0000 7fc2ce46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:45.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:45.127+0000 7fc2ce46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:45.592 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:45.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:45.722+0000 7fadf9e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:45.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:45.727+0000 7fadf9e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:45.974 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_1' created 2024-01-11T10:11:45.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-01-11T10:11:46.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:46.126+0000 7ffa10b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:46.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:46.140+0000 7ffa10b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:46.980 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-01-11T10:11:47.400 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:47.400 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:47.401 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:11:47.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:11:47.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.521+0000 7f4f5431e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.529+0000 7f4f5431e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:47.525 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:47.526 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:47.529 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:47.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:11:47.636 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.647+0000 7f356199e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.655+0000 7f356199e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:47.651 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:47.652 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:47.654 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:47.654 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:11:47.762 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.773+0000 7ff9b5f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.764 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.781+0000 7ff9b5f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:47.778 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:47.779 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:47.781 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:47.781 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:11:47.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:47.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.902+0000 7f84ae99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.894 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.910+0000 7f84ae99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.908 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:11:47.908 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:11:47.908 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:11:47.908 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:11:47.908 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:11:47.908 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:11:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:11:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:11:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:11:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:11:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:11:47.909 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:11:47.911 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:11:47.911 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:11:47.931 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.946+0000 7fbc4635e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.934 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:47.950+0000 7fbc4635e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:47.999 DEBUG:teuthology.orchestra.run.smithi040:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-01-11T10:11:48.001 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-01-11T10:11:48.002 INFO:teuthology.orchestra.run:waiting for 3000 2024-01-11T10:11:48.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:hints = 1 2024-01-11T10:11:48.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-01-11T10:11:48.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object prefix: benchmark_data_smithi040_39286 2024-01-11T10:11:48.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:11:48.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 0 0 0 0 0 0 - 0 2024-01-11T10:11:49.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1 16 187 171 10.6869 10.6875 0.13791 0.0860254 2024-01-11T10:11:50.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:50.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2 16 359 343 10.718 10.75 0.149374 0.0895552 2024-01-11T10:11:50.127 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:50.142+0000 7f631ca60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:50.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:50.147+0000 7f631ca60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:51.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3 16 519 503 10.4784 10 0.169783 0.0932752 2024-01-11T10:11:52.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 4 16 701 685 10.7023 11.375 0.107346 0.0920133 2024-01-11T10:11:52.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:52.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:52.405+0000 7f10ec51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:52.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:52.411+0000 7f10ec51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:52.912 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:11:52.913 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:11:53.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:53.059+0000 7f3d3cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:53.054 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:53.070+0000 7f3d3cead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:53.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 5 16 865 849 10.6116 10.25 0.129687 0.0932538 2024-01-11T10:11:53.999 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:11:54.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:11:54.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 6 16 1041 1025 10.6762 11 0.124208 0.0930422 2024-01-11T10:11:54.135 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:54.144+0000 7fa8d495e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:54.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:54.155+0000 7fa8d495e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:54.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:54.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:54.705+0000 7f5add66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:54.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:54.710+0000 7f5add66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:55.004 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:11:55.024 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:11:55.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 7 16 1207 1191 10.633 10.375 0.15954 0.0929837 2024-01-11T10:11:55.157 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:55.167+0000 7fe2f2f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:55.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:55.176+0000 7fe2f2f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:56.010 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:11:56.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:11:56.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 8 16 1366 1350 10.546 9.9375 0.165541 0.0938315 2024-01-11T10:11:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:56.187+0000 7f12a9417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:56.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:56.197+0000 7f12a9417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:56.901 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:11:57.013 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:11:57.014 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:57.028+0000 7f1c7082d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:57.019 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:57.036+0000 7f1c7082d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:57.032 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:11:57.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:57.056 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 9 16 1499 1483 10.2978 8.3125 0.0501219 0.0963673 2024-01-11T10:11:57.152 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:57.164+0000 7f91fca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:57.155 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:57.172+0000 7f91fca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:57.528 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:57.529 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:57.547 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-01-11T10:11:57.031408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":52888096,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":935,"num_write_kb":52966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":249784,"kb_used_data":9240,"kb_used_omap":0,"kb_used_meta":240512,"kb_avail":377237576,"statfs":{"total":386547056640,"available":386291277824,"internally_reserved":0,"allocated":9461760,"data_stored":136851376,"data_compressed":6488898,"data_compressed_allocated":8548352,"data_compressed_original":136527872,"omap_allocated":0,"internal_metadata":246284288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.896666"},"pg_stats":[{"pgid":"4.f","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.900572+0000","last_change":"2024-01-11T10:11:47.138050+0000","last_active":"2024-01-11T10:11:52.900572+0000","last_peered":"2024-01-11T10:11:52.900572+0000","last_clean":"2024-01-11T10:11:52.900572+0000","last_became_active":"2024-01-11T10:11:47.134738+0000","last_became_peered":"2024-01-11T10:11:47.134738+0000","last_unstale":"2024-01-11T10:11:52.900572+0000","last_undegraded":"2024-01-11T10:11:52.900572+0000","last_fullsized":"2024-01-11T10:11:52.900572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"25'48","reported_seq":60,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.963026+0000","last_change":"2024-01-11T10:11:47.125437+0000","last_active":"2024-01-11T10:11:52.963026+0000","last_peered":"2024-01-11T10:11:52.963026+0000","last_clean":"2024-01-11T10:11:52.963026+0000","last_became_active":"2024-01-11T10:11:47.122491+0000","last_became_peered":"2024-01-11T10:11:47.122491+0000","last_unstale":"2024-01-11T10:11:52.963026+0000","last_undegraded":"2024-01-11T10:11:52.963026+0000","last_fullsized":"2024-01-11T10:11:52.963026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"25'53","reported_seq":65,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.135290+0000","last_change":"2024-01-11T10:11:47.235698+0000","last_active":"2024-01-11T10:11:52.135290+0000","last_peered":"2024-01-11T10:11:52.135290+0000","last_clean":"2024-01-11T10:11:52.135290+0000","last_became_active":"2024-01-11T10:11:47.229122+0000","last_became_peered":"2024-01-11T10:11:47.229122+0000","last_unstale":"2024-01-11T10:11:52.135290+0000","last_undegraded":"2024-01-11T10:11:52.135290+0000","last_fullsized":"2024-01-11T10:11:52.135290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"25'48","reported_seq":60,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.988549+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:11:51.988549+0000","last_peered":"2024-01-11T10:11:51.988549+0000","last_clean":"2024-01-11T10:11:51.988549+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:11:51.988549+0000","last_undegraded":"2024-01-11T10:11:51.988549+0000","last_fullsized":"2024-01-11T10:11:51.988549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"25'53","reported_seq":65,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.105856+0000","last_change":"2024-01-11T10:11:47.214794+0000","last_active":"2024-01-11T10:11:53.105856+0000","last_peered":"2024-01-11T10:11:53.105856+0000","last_clean":"2024-01-11T10:11:53.105856+0000","last_became_active":"2024-01-11T10:11:47.211625+0000","last_became_peered":"2024-01-11T10:11:47.211625+0000","last_unstale":"2024-01-11T10:11:53.105856+0000","last_undegraded":"2024-01-11T10:11:53.105856+0000","last_fullsized":"2024-01-11T10:11:53.105856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"25'43","reported_seq":55,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.005593+0000","last_change":"2024-01-11T10:11:47.281474+0000","last_active":"2024-01-11T10:11:52.005593+0000","last_peered":"2024-01-11T10:11:52.005593+0000","last_clean":"2024-01-11T10:11:52.005593+0000","last_became_active":"2024-01-11T10:11:47.278218+0000","last_became_peered":"2024-01-11T10:11:47.278218+0000","last_unstale":"2024-01-11T10:11:52.005593+0000","last_undegraded":"2024-01-11T10:11:52.005593+0000","last_fullsized":"2024-01-11T10:11:52.005593+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"25'46","reported_seq":58,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.984858+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:11:51.984858+0000","last_peered":"2024-01-11T10:11:51.984858+0000","last_clean":"2024-01-11T10:11:51.984858+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:11:51.984858+0000","last_undegraded":"2024-01-11T10:11:51.984858+0000","last_fullsized":"2024-01-11T10:11:51.984858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"25'41","reported_seq":53,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.871246+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:11:51.871246+0000","last_peered":"2024-01-11T10:11:51.871246+0000","last_clean":"2024-01-11T10:11:51.871246+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:11:51.871246+0000","last_undegraded":"2024-01-11T10:11:51.871246+0000","last_fullsized":"2024-01-11T10:11:51.871246+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"25'42","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.044684+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:11:52.044684+0000","last_peered":"2024-01-11T10:11:52.044684+0000","last_clean":"2024-01-11T10:11:52.044684+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:11:52.044684+0000","last_undegraded":"2024-01-11T10:11:52.044684+0000","last_fullsized":"2024-01-11T10:11:52.044684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":164,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.125614+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:11:48.125614+0000","last_peered":"2024-01-11T10:11:48.125614+0000","last_clean":"2024-01-11T10:11:48.125614+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:11:48.125614+0000","last_undegraded":"2024-01-11T10:11:48.125614+0000","last_fullsized":"2024-01-11T10:11:48.125614+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.117088+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:11:48.117088+0000","last_peered":"2024-01-11T10:11:48.117088+0000","last_clean":"2024-01-11T10:11:48.117088+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:11:48.117088+0000","last_undegraded":"2024-01-11T10:11:48.117088+0000","last_fullsized":"2024-01-11T10:11:48.117088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"25'45","reported_seq":57,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.083055+0000","last_change":"2024-01-11T10:11:47.198675+0000","last_active":"2024-01-11T10:11:53.083055+0000","last_peered":"2024-01-11T10:11:53.083055+0000","last_clean":"2024-01-11T10:11:53.083055+0000","last_became_active":"2024-01-11T10:11:47.194485+0000","last_became_peered":"2024-01-11T10:11:47.194485+0000","last_unstale":"2024-01-11T10:11:53.083055+0000","last_undegraded":"2024-01-11T10:11:53.083055+0000","last_fullsized":"2024-01-11T10:11:53.083055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.016569+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:11:48.016569+0000","last_peered":"2024-01-11T10:11:48.016569+0000","last_clean":"2024-01-11T10:11:48.016569+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:11:48.016569+0000","last_undegraded":"2024-01-11T10:11:48.016569+0000","last_fullsized":"2024-01-11T10:11:48.016569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"25'58","reported_seq":70,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.082749+0000","last_change":"2024-01-11T10:11:47.157444+0000","last_active":"2024-01-11T10:11:53.082749+0000","last_peered":"2024-01-11T10:11:53.082749+0000","last_clean":"2024-01-11T10:11:53.082749+0000","last_became_active":"2024-01-11T10:11:47.154600+0000","last_became_peered":"2024-01-11T10:11:47.154600+0000","last_unstale":"2024-01-11T10:11:53.082749+0000","last_undegraded":"2024-01-11T10:11:53.082749+0000","last_fullsized":"2024-01-11T10:11:53.082749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.117534+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:11:48.117534+0000","last_peered":"2024-01-11T10:11:48.117534+0000","last_clean":"2024-01-11T10:11:48.117534+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:11:48.117534+0000","last_undegraded":"2024-01-11T10:11:48.117534+0000","last_fullsized":"2024-01-11T10:11:48.117534+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.978338+0000","last_change":"2024-01-11T10:11:47.151805+0000","last_active":"2024-01-11T10:11:52.978338+0000","last_peered":"2024-01-11T10:11:52.978338+0000","last_clean":"2024-01-11T10:11:52.978338+0000","last_became_active":"2024-01-11T10:11:47.149100+0000","last_became_peered":"2024-01-11T10:11:47.149100+0000","last_unstale":"2024-01-11T10:11:52.978338+0000","last_undegraded":"2024-01-11T10:11:52.978338+0000","last_fullsized":"2024-01-11T10:11:52.978338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.015562+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:11:48.015562+0000","last_peered":"2024-01-11T10:11:48.015562+0000","last_clean":"2024-01-11T10:11:48.015562+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:11:48.015562+0000","last_undegraded":"2024-01-11T10:11:48.015562+0000","last_fullsized":"2024-01-11T10:11:48.015562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"25'50","reported_seq":62,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.147955+0000","last_change":"2024-01-11T10:11:47.206076+0000","last_active":"2024-01-11T10:11:52.147955+0000","last_peered":"2024-01-11T10:11:52.147955+0000","last_clean":"2024-01-11T10:11:52.147955+0000","last_became_active":"2024-01-11T10:11:47.203256+0000","last_became_peered":"2024-01-11T10:11:47.203256+0000","last_unstale":"2024-01-11T10:11:52.147955+0000","last_undegraded":"2024-01-11T10:11:52.147955+0000","last_fullsized":"2024-01-11T10:11:52.147955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.100828+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:11:48.100828+0000","last_peered":"2024-01-11T10:11:48.100828+0000","last_clean":"2024-01-11T10:11:48.100828+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:11:48.100828+0000","last_undegraded":"2024-01-11T10:11:48.100828+0000","last_fullsized":"2024-01-11T10:11:48.100828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"25'58","reported_seq":70,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.153684+0000","last_change":"2024-01-11T10:11:47.209065+0000","last_active":"2024-01-11T10:11:53.153684+0000","last_peered":"2024-01-11T10:11:53.153684+0000","last_clean":"2024-01-11T10:11:53.153684+0000","last_became_active":"2024-01-11T10:11:47.196382+0000","last_became_peered":"2024-01-11T10:11:47.196382+0000","last_unstale":"2024-01-11T10:11:53.153684+0000","last_undegraded":"2024-01-11T10:11:53.153684+0000","last_fullsized":"2024-01-11T10:11:53.153684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.132257+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:11:48.132257+0000","last_peered":"2024-01-11T10:11:48.132257+0000","last_clean":"2024-01-11T10:11:48.132257+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:11:48.132257+0000","last_undegraded":"2024-01-11T10:11:48.132257+0000","last_fullsized":"2024-01-11T10:11:48.132257+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.732486+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:11:52.732486+0000","last_peered":"2024-01-11T10:11:52.732486+0000","last_clean":"2024-01-11T10:11:52.732486+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:11:52.732486+0000","last_undegraded":"2024-01-11T10:11:52.732486+0000","last_fullsized":"2024-01-11T10:11:52.732486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.091671+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:11:48.091671+0000","last_peered":"2024-01-11T10:11:48.091671+0000","last_clean":"2024-01-11T10:11:48.091671+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:11:48.091671+0000","last_undegraded":"2024-01-11T10:11:48.091671+0000","last_fullsized":"2024-01-11T10:11:48.091671+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"25'44","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.849256+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:11:51.849256+0000","last_peered":"2024-01-11T10:11:51.849256+0000","last_clean":"2024-01-11T10:11:51.849256+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:11:51.849256+0000","last_undegraded":"2024-01-11T10:11:51.849256+0000","last_fullsized":"2024-01-11T10:11:51.849256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.118362+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:11:48.118362+0000","last_peered":"2024-01-11T10:11:48.118362+0000","last_clean":"2024-01-11T10:11:48.118362+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:11:48.118362+0000","last_undegraded":"2024-01-11T10:11:48.118362+0000","last_fullsized":"2024-01-11T10:11:48.118362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":798,"num_write_kb":51072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51236,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94320604,"statfs":{"total":96636764160,"available":96584298496,"internally_reserved":0,"allocated":2256896,"data_stored":32377972,"data_compressed":1535622,"data_compressed_allocated":2023424,"data_compressed_original":32292864,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55776,"kb_used_data":2584,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":2646016,"data_stored":38603892,"data_compressed":1831150,"data_compressed_allocated":2412544,"data_compressed_original":38518784,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65164,"kb_used_data":2244,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2297856,"data_stored":33033332,"data_compressed":1566708,"data_compressed_allocated":2064384,"data_compressed_original":32948224,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77608,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94294232,"statfs":{"total":96636764160,"available":96557293568,"internally_reserved":0,"allocated":2260992,"data_stored":32836180,"data_compressed":1555418,"data_compressed_allocated":2048000,"data_compressed_original":32768000,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:57.548 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:57.667 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:57.679+0000 7f1aab75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:57.670 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:57.687+0000 7f1aab75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:57.901 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:57.902 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:57.917 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-01-11T10:11:57.031408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":52888096,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":935,"num_write_kb":52966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":879,"ondisk_log_size":879,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":249784,"kb_used_data":9240,"kb_used_omap":0,"kb_used_meta":240512,"kb_avail":377237576,"statfs":{"total":386547056640,"available":386291277824,"internally_reserved":0,"allocated":9461760,"data_stored":136851376,"data_compressed":6488898,"data_compressed_allocated":8548352,"data_compressed_original":136527872,"omap_allocated":0,"internal_metadata":246284288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.896666"},"pg_stats":[{"pgid":"4.f","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.900572+0000","last_change":"2024-01-11T10:11:47.138050+0000","last_active":"2024-01-11T10:11:52.900572+0000","last_peered":"2024-01-11T10:11:52.900572+0000","last_clean":"2024-01-11T10:11:52.900572+0000","last_became_active":"2024-01-11T10:11:47.134738+0000","last_became_peered":"2024-01-11T10:11:47.134738+0000","last_unstale":"2024-01-11T10:11:52.900572+0000","last_undegraded":"2024-01-11T10:11:52.900572+0000","last_fullsized":"2024-01-11T10:11:52.900572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"25'48","reported_seq":60,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.963026+0000","last_change":"2024-01-11T10:11:47.125437+0000","last_active":"2024-01-11T10:11:52.963026+0000","last_peered":"2024-01-11T10:11:52.963026+0000","last_clean":"2024-01-11T10:11:52.963026+0000","last_became_active":"2024-01-11T10:11:47.122491+0000","last_became_peered":"2024-01-11T10:11:47.122491+0000","last_unstale":"2024-01-11T10:11:52.963026+0000","last_undegraded":"2024-01-11T10:11:52.963026+0000","last_fullsized":"2024-01-11T10:11:52.963026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"25'53","reported_seq":65,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.135290+0000","last_change":"2024-01-11T10:11:47.235698+0000","last_active":"2024-01-11T10:11:52.135290+0000","last_peered":"2024-01-11T10:11:52.135290+0000","last_clean":"2024-01-11T10:11:52.135290+0000","last_became_active":"2024-01-11T10:11:47.229122+0000","last_became_peered":"2024-01-11T10:11:47.229122+0000","last_unstale":"2024-01-11T10:11:52.135290+0000","last_undegraded":"2024-01-11T10:11:52.135290+0000","last_fullsized":"2024-01-11T10:11:52.135290+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"25'48","reported_seq":60,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.988549+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:11:51.988549+0000","last_peered":"2024-01-11T10:11:51.988549+0000","last_clean":"2024-01-11T10:11:51.988549+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:11:51.988549+0000","last_undegraded":"2024-01-11T10:11:51.988549+0000","last_fullsized":"2024-01-11T10:11:51.988549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"25'53","reported_seq":65,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.105856+0000","last_change":"2024-01-11T10:11:47.214794+0000","last_active":"2024-01-11T10:11:53.105856+0000","last_peered":"2024-01-11T10:11:53.105856+0000","last_clean":"2024-01-11T10:11:53.105856+0000","last_became_active":"2024-01-11T10:11:47.211625+0000","last_became_peered":"2024-01-11T10:11:47.211625+0000","last_unstale":"2024-01-11T10:11:53.105856+0000","last_undegraded":"2024-01-11T10:11:53.105856+0000","last_fullsized":"2024-01-11T10:11:53.105856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"25'43","reported_seq":55,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.005593+0000","last_change":"2024-01-11T10:11:47.281474+0000","last_active":"2024-01-11T10:11:52.005593+0000","last_peered":"2024-01-11T10:11:52.005593+0000","last_clean":"2024-01-11T10:11:52.005593+0000","last_became_active":"2024-01-11T10:11:47.278218+0000","last_became_peered":"2024-01-11T10:11:47.278218+0000","last_unstale":"2024-01-11T10:11:52.005593+0000","last_undegraded":"2024-01-11T10:11:52.005593+0000","last_fullsized":"2024-01-11T10:11:52.005593+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2883584,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":2816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"25'46","reported_seq":58,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.984858+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:11:51.984858+0000","last_peered":"2024-01-11T10:11:51.984858+0000","last_clean":"2024-01-11T10:11:51.984858+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:11:51.984858+0000","last_undegraded":"2024-01-11T10:11:51.984858+0000","last_fullsized":"2024-01-11T10:11:51.984858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3080192,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"25'41","reported_seq":53,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.871246+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:11:51.871246+0000","last_peered":"2024-01-11T10:11:51.871246+0000","last_clean":"2024-01-11T10:11:51.871246+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:11:51.871246+0000","last_undegraded":"2024-01-11T10:11:51.871246+0000","last_fullsized":"2024-01-11T10:11:51.871246+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":42,"num_write_kb":2688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"25'42","reported_seq":54,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.044684+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:11:52.044684+0000","last_peered":"2024-01-11T10:11:52.044684+0000","last_clean":"2024-01-11T10:11:52.044684+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:11:52.044684+0000","last_undegraded":"2024-01-11T10:11:52.044684+0000","last_fullsized":"2024-01-11T10:11:52.044684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2818048,"num_objects":43,"num_object_clones":0,"num_object_copies":129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43,"num_write_kb":2752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":164,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.125614+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:11:48.125614+0000","last_peered":"2024-01-11T10:11:48.125614+0000","last_clean":"2024-01-11T10:11:48.125614+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:11:48.125614+0000","last_undegraded":"2024-01-11T10:11:48.125614+0000","last_fullsized":"2024-01-11T10:11:48.125614+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.117088+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:11:48.117088+0000","last_peered":"2024-01-11T10:11:48.117088+0000","last_clean":"2024-01-11T10:11:48.117088+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:11:48.117088+0000","last_undegraded":"2024-01-11T10:11:48.117088+0000","last_fullsized":"2024-01-11T10:11:48.117088+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"25'45","reported_seq":57,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.083055+0000","last_change":"2024-01-11T10:11:47.198675+0000","last_active":"2024-01-11T10:11:53.083055+0000","last_peered":"2024-01-11T10:11:53.083055+0000","last_clean":"2024-01-11T10:11:53.083055+0000","last_became_active":"2024-01-11T10:11:47.194485+0000","last_became_peered":"2024-01-11T10:11:47.194485+0000","last_unstale":"2024-01-11T10:11:53.083055+0000","last_undegraded":"2024-01-11T10:11:53.083055+0000","last_fullsized":"2024-01-11T10:11:53.083055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.016569+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:11:48.016569+0000","last_peered":"2024-01-11T10:11:48.016569+0000","last_clean":"2024-01-11T10:11:48.016569+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:11:48.016569+0000","last_undegraded":"2024-01-11T10:11:48.016569+0000","last_fullsized":"2024-01-11T10:11:48.016569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"25'58","reported_seq":70,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.082749+0000","last_change":"2024-01-11T10:11:47.157444+0000","last_active":"2024-01-11T10:11:53.082749+0000","last_peered":"2024-01-11T10:11:53.082749+0000","last_clean":"2024-01-11T10:11:53.082749+0000","last_became_active":"2024-01-11T10:11:47.154600+0000","last_became_peered":"2024-01-11T10:11:47.154600+0000","last_unstale":"2024-01-11T10:11:53.082749+0000","last_undegraded":"2024-01-11T10:11:53.082749+0000","last_fullsized":"2024-01-11T10:11:53.082749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.117534+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:11:48.117534+0000","last_peered":"2024-01-11T10:11:48.117534+0000","last_clean":"2024-01-11T10:11:48.117534+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:11:48.117534+0000","last_undegraded":"2024-01-11T10:11:48.117534+0000","last_fullsized":"2024-01-11T10:11:48.117534+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.978338+0000","last_change":"2024-01-11T10:11:47.151805+0000","last_active":"2024-01-11T10:11:52.978338+0000","last_peered":"2024-01-11T10:11:52.978338+0000","last_clean":"2024-01-11T10:11:52.978338+0000","last_became_active":"2024-01-11T10:11:47.149100+0000","last_became_peered":"2024-01-11T10:11:47.149100+0000","last_unstale":"2024-01-11T10:11:52.978338+0000","last_undegraded":"2024-01-11T10:11:52.978338+0000","last_fullsized":"2024-01-11T10:11:52.978338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.015562+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:11:48.015562+0000","last_peered":"2024-01-11T10:11:48.015562+0000","last_clean":"2024-01-11T10:11:48.015562+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:11:48.015562+0000","last_undegraded":"2024-01-11T10:11:48.015562+0000","last_fullsized":"2024-01-11T10:11:48.015562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"25'50","reported_seq":62,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.147955+0000","last_change":"2024-01-11T10:11:47.206076+0000","last_active":"2024-01-11T10:11:52.147955+0000","last_peered":"2024-01-11T10:11:52.147955+0000","last_clean":"2024-01-11T10:11:52.147955+0000","last_became_active":"2024-01-11T10:11:47.203256+0000","last_became_peered":"2024-01-11T10:11:47.203256+0000","last_unstale":"2024-01-11T10:11:52.147955+0000","last_undegraded":"2024-01-11T10:11:52.147955+0000","last_fullsized":"2024-01-11T10:11:52.147955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51,"num_write_kb":3264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.100828+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:11:48.100828+0000","last_peered":"2024-01-11T10:11:48.100828+0000","last_clean":"2024-01-11T10:11:48.100828+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:11:48.100828+0000","last_undegraded":"2024-01-11T10:11:48.100828+0000","last_fullsized":"2024-01-11T10:11:48.100828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"25'58","reported_seq":70,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.153684+0000","last_change":"2024-01-11T10:11:47.209065+0000","last_active":"2024-01-11T10:11:53.153684+0000","last_peered":"2024-01-11T10:11:53.153684+0000","last_clean":"2024-01-11T10:11:53.153684+0000","last_became_active":"2024-01-11T10:11:47.196382+0000","last_became_peered":"2024-01-11T10:11:47.196382+0000","last_unstale":"2024-01-11T10:11:53.153684+0000","last_undegraded":"2024-01-11T10:11:53.153684+0000","last_fullsized":"2024-01-11T10:11:53.153684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":33,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.132257+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:11:48.132257+0000","last_peered":"2024-01-11T10:11:48.132257+0000","last_clean":"2024-01-11T10:11:48.132257+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:11:48.132257+0000","last_undegraded":"2024-01-11T10:11:48.132257+0000","last_fullsized":"2024-01-11T10:11:48.132257+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.732486+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:11:52.732486+0000","last_peered":"2024-01-11T10:11:52.732486+0000","last_clean":"2024-01-11T10:11:52.732486+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:11:52.732486+0000","last_undegraded":"2024-01-11T10:11:52.732486+0000","last_fullsized":"2024-01-11T10:11:52.732486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.091671+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:11:48.091671+0000","last_peered":"2024-01-11T10:11:48.091671+0000","last_clean":"2024-01-11T10:11:48.091671+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:11:48.091671+0000","last_undegraded":"2024-01-11T10:11:48.091671+0000","last_fullsized":"2024-01-11T10:11:48.091671+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"25'44","reported_seq":56,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:51.849256+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:11:51.849256+0000","last_peered":"2024-01-11T10:11:51.849256+0000","last_clean":"2024-01-11T10:11:51.849256+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:11:51.849256+0000","last_undegraded":"2024-01-11T10:11:51.849256+0000","last_fullsized":"2024-01-11T10:11:51.849256+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2949120,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":2880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":29,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.118362+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:11:48.118362+0000","last_peered":"2024-01-11T10:11:48.118362+0000","last_clean":"2024-01-11T10:11:48.118362+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:11:48.118362+0000","last_undegraded":"2024-01-11T10:11:48.118362+0000","last_fullsized":"2024-01-11T10:11:48.118362+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":52297728,"num_objects":798,"num_object_clones":0,"num_object_copies":2394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":798,"num_write_kb":51072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51236,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94320604,"statfs":{"total":96636764160,"available":96584298496,"internally_reserved":0,"allocated":2256896,"data_stored":32377972,"data_compressed":1535622,"data_compressed_allocated":2023424,"data_compressed_original":32292864,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55776,"kb_used_data":2584,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":2646016,"data_stored":38603892,"data_compressed":1831150,"data_compressed_allocated":2412544,"data_compressed_original":38518784,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":65164,"kb_used_data":2244,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94306676,"statfs":{"total":96636764160,"available":96570036224,"internally_reserved":0,"allocated":2297856,"data_stored":33033332,"data_compressed":1566708,"data_compressed_allocated":2064384,"data_compressed_original":32948224,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":77608,"kb_used_data":2208,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94294232,"statfs":{"total":96636764160,"available":96557293568,"internally_reserved":0,"allocated":2260992,"data_stored":32836180,"data_compressed":1555418,"data_compressed_allocated":2048000,"data_compressed_original":32768000,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:57.918 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:11:57.918 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:11:57.918 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:11:58.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:58.056+0000 7fc30ab1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:58.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:58.064+0000 7fc30ab1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:58.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 10 16 1650 1634 10.2117 9.4375 0.140151 0.097269 2024-01-11T10:11:58.244 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:11:58.244 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:11:58.262 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":130,"stamp":"2024-01-11T10:11:58.135815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":79430176,"num_objects":1205,"num_object_clones":0,"num_object_copies":3615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1340,"num_write_kb":78886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1285,"ondisk_log_size":1285,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":246796,"kb_used_data":14572,"kb_used_omap":0,"kb_used_meta":232192,"kb_avail":377240564,"statfs":{"total":386547056640,"available":386294337536,"internally_reserved":0,"allocated":14921728,"data_stored":223571729,"data_compressed":10605444,"data_compressed_allocated":13967360,"data_compressed_original":223232000,"omap_allocated":0,"internal_metadata":237764608},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26542080,"num_objects":405,"num_object_clones":0,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":405,"num_write_kb":25920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.116983"},"pg_stats":[{"pgid":"4.f","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.900572+0000","last_change":"2024-01-11T10:11:47.138050+0000","last_active":"2024-01-11T10:11:52.900572+0000","last_peered":"2024-01-11T10:11:52.900572+0000","last_clean":"2024-01-11T10:11:52.900572+0000","last_became_active":"2024-01-11T10:11:47.134738+0000","last_became_peered":"2024-01-11T10:11:47.134738+0000","last_unstale":"2024-01-11T10:11:52.900572+0000","last_undegraded":"2024-01-11T10:11:52.900572+0000","last_fullsized":"2024-01-11T10:11:52.900572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"25'48","reported_seq":60,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.963026+0000","last_change":"2024-01-11T10:11:47.125437+0000","last_active":"2024-01-11T10:11:52.963026+0000","last_peered":"2024-01-11T10:11:52.963026+0000","last_clean":"2024-01-11T10:11:52.963026+0000","last_became_active":"2024-01-11T10:11:47.122491+0000","last_became_peered":"2024-01-11T10:11:47.122491+0000","last_unstale":"2024-01-11T10:11:52.963026+0000","last_undegraded":"2024-01-11T10:11:52.963026+0000","last_fullsized":"2024-01-11T10:11:52.963026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":3136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"27'102","reported_seq":116,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.037441+0000","last_change":"2024-01-11T10:11:47.235698+0000","last_active":"2024-01-11T10:11:57.037441+0000","last_peered":"2024-01-11T10:11:57.037441+0000","last_clean":"2024-01-11T10:11:57.037441+0000","last_became_active":"2024-01-11T10:11:47.229122+0000","last_became_peered":"2024-01-11T10:11:47.229122+0000","last_unstale":"2024-01-11T10:11:57.037441+0000","last_undegraded":"2024-01-11T10:11:57.037441+0000","last_fullsized":"2024-01-11T10:11:57.037441+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6750208,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":6592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"28'87","reported_seq":104,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.053448+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:11:57.053448+0000","last_peered":"2024-01-11T10:11:57.053448+0000","last_clean":"2024-01-11T10:11:57.053448+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:11:57.053448+0000","last_undegraded":"2024-01-11T10:11:57.053448+0000","last_fullsized":"2024-01-11T10:11:57.053448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"25'53","reported_seq":65,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.105856+0000","last_change":"2024-01-11T10:11:47.214794+0000","last_active":"2024-01-11T10:11:53.105856+0000","last_peered":"2024-01-11T10:11:53.105856+0000","last_clean":"2024-01-11T10:11:53.105856+0000","last_became_active":"2024-01-11T10:11:47.211625+0000","last_became_peered":"2024-01-11T10:11:47.211625+0000","last_unstale":"2024-01-11T10:11:53.105856+0000","last_undegraded":"2024-01-11T10:11:53.105856+0000","last_fullsized":"2024-01-11T10:11:53.105856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3538944,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":54,"num_write_kb":3456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"27'91","reported_seq":105,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.983449+0000","last_change":"2024-01-11T10:11:47.281474+0000","last_active":"2024-01-11T10:11:56.983449+0000","last_peered":"2024-01-11T10:11:56.983449+0000","last_clean":"2024-01-11T10:11:56.983449+0000","last_became_active":"2024-01-11T10:11:47.278218+0000","last_became_peered":"2024-01-11T10:11:47.278218+0000","last_unstale":"2024-01-11T10:11:56.983449+0000","last_undegraded":"2024-01-11T10:11:56.983449+0000","last_fullsized":"2024-01-11T10:11:56.983449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":5888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"27'101","reported_seq":115,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.072851+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:11:57.072851+0000","last_peered":"2024-01-11T10:11:57.072851+0000","last_clean":"2024-01-11T10:11:57.072851+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:11:57.072851+0000","last_undegraded":"2024-01-11T10:11:57.072851+0000","last_fullsized":"2024-01-11T10:11:57.072851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"27'94","reported_seq":108,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.962228+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:11:56.962228+0000","last_peered":"2024-01-11T10:11:56.962228+0000","last_clean":"2024-01-11T10:11:56.962228+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:11:56.962228+0000","last_undegraded":"2024-01-11T10:11:56.962228+0000","last_fullsized":"2024-01-11T10:11:56.962228+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":6080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"27'103","reported_seq":117,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.897844+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:11:56.897844+0000","last_peered":"2024-01-11T10:11:56.897844+0000","last_clean":"2024-01-11T10:11:56.897844+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:11:56.897844+0000","last_undegraded":"2024-01-11T10:11:56.897844+0000","last_fullsized":"2024-01-11T10:11:56.897844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":170,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.076470+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:11:57.076470+0000","last_peered":"2024-01-11T10:11:57.076470+0000","last_clean":"2024-01-11T10:11:57.076470+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:11:57.076470+0000","last_undegraded":"2024-01-11T10:11:57.076470+0000","last_fullsized":"2024-01-11T10:11:57.076470+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.214392+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:11:55.214392+0000","last_peered":"2024-01-11T10:11:55.214392+0000","last_clean":"2024-01-11T10:11:55.214392+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:11:55.214392+0000","last_undegraded":"2024-01-11T10:11:55.214392+0000","last_fullsized":"2024-01-11T10:11:55.214392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"25'45","reported_seq":57,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.083055+0000","last_change":"2024-01-11T10:11:47.198675+0000","last_active":"2024-01-11T10:11:53.083055+0000","last_peered":"2024-01-11T10:11:53.083055+0000","last_clean":"2024-01-11T10:11:53.083055+0000","last_became_active":"2024-01-11T10:11:47.194485+0000","last_became_peered":"2024-01-11T10:11:47.194485+0000","last_unstale":"2024-01-11T10:11:53.083055+0000","last_undegraded":"2024-01-11T10:11:53.083055+0000","last_fullsized":"2024-01-11T10:11:53.083055+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3014656,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":46,"num_write_kb":2944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.016569+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:11:48.016569+0000","last_peered":"2024-01-11T10:11:48.016569+0000","last_clean":"2024-01-11T10:11:48.016569+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:11:48.016569+0000","last_undegraded":"2024-01-11T10:11:48.016569+0000","last_fullsized":"2024-01-11T10:11:48.016569+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"25'58","reported_seq":70,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.082749+0000","last_change":"2024-01-11T10:11:47.157444+0000","last_active":"2024-01-11T10:11:53.082749+0000","last_peered":"2024-01-11T10:11:53.082749+0000","last_clean":"2024-01-11T10:11:53.082749+0000","last_became_active":"2024-01-11T10:11:47.154600+0000","last_became_peered":"2024-01-11T10:11:47.154600+0000","last_unstale":"2024-01-11T10:11:53.082749+0000","last_undegraded":"2024-01-11T10:11:53.082749+0000","last_fullsized":"2024-01-11T10:11:53.082749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.177405+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:11:55.177405+0000","last_peered":"2024-01-11T10:11:55.177405+0000","last_clean":"2024-01-11T10:11:55.177405+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:11:55.177405+0000","last_undegraded":"2024-01-11T10:11:55.177405+0000","last_fullsized":"2024-01-11T10:11:55.177405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.978338+0000","last_change":"2024-01-11T10:11:47.151805+0000","last_active":"2024-01-11T10:11:52.978338+0000","last_peered":"2024-01-11T10:11:52.978338+0000","last_clean":"2024-01-11T10:11:52.978338+0000","last_became_active":"2024-01-11T10:11:47.149100+0000","last_became_peered":"2024-01-11T10:11:47.149100+0000","last_unstale":"2024-01-11T10:11:52.978338+0000","last_undegraded":"2024-01-11T10:11:52.978338+0000","last_fullsized":"2024-01-11T10:11:52.978338+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.015562+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:11:48.015562+0000","last_peered":"2024-01-11T10:11:48.015562+0000","last_clean":"2024-01-11T10:11:48.015562+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:11:48.015562+0000","last_undegraded":"2024-01-11T10:11:48.015562+0000","last_fullsized":"2024-01-11T10:11:48.015562+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"27'99","reported_seq":113,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.040994+0000","last_change":"2024-01-11T10:11:47.206076+0000","last_active":"2024-01-11T10:11:57.040994+0000","last_peered":"2024-01-11T10:11:57.040994+0000","last_clean":"2024-01-11T10:11:57.040994+0000","last_became_active":"2024-01-11T10:11:47.203256+0000","last_became_peered":"2024-01-11T10:11:47.203256+0000","last_unstale":"2024-01-11T10:11:57.040994+0000","last_undegraded":"2024-01-11T10:11:57.040994+0000","last_fullsized":"2024-01-11T10:11:57.040994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.100828+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:11:48.100828+0000","last_peered":"2024-01-11T10:11:48.100828+0000","last_clean":"2024-01-11T10:11:48.100828+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:11:48.100828+0000","last_undegraded":"2024-01-11T10:11:48.100828+0000","last_fullsized":"2024-01-11T10:11:48.100828+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"25'58","reported_seq":70,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:53.153684+0000","last_change":"2024-01-11T10:11:47.209065+0000","last_active":"2024-01-11T10:11:53.153684+0000","last_peered":"2024-01-11T10:11:53.153684+0000","last_clean":"2024-01-11T10:11:53.153684+0000","last_became_active":"2024-01-11T10:11:47.196382+0000","last_became_peered":"2024-01-11T10:11:47.196382+0000","last_unstale":"2024-01-11T10:11:53.153684+0000","last_undegraded":"2024-01-11T10:11:53.153684+0000","last_fullsized":"2024-01-11T10:11:53.153684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":59,"num_write_kb":3776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.077614+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:11:57.077614+0000","last_peered":"2024-01-11T10:11:57.077614+0000","last_clean":"2024-01-11T10:11:57.077614+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:11:57.077614+0000","last_undegraded":"2024-01-11T10:11:57.077614+0000","last_fullsized":"2024-01-11T10:11:57.077614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"25'51","reported_seq":63,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:52.732486+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:11:52.732486+0000","last_peered":"2024-01-11T10:11:52.732486+0000","last_clean":"2024-01-11T10:11:52.732486+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:11:52.732486+0000","last_undegraded":"2024-01-11T10:11:52.732486+0000","last_fullsized":"2024-01-11T10:11:52.732486+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52,"num_write_kb":3328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":27,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-01-11T10:11:48.091671+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:11:48.091671+0000","last_peered":"2024-01-11T10:11:48.091671+0000","last_clean":"2024-01-11T10:11:48.091671+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:11:48.091671+0000","last_undegraded":"2024-01-11T10:11:48.091671+0000","last_fullsized":"2024-01-11T10:11:48.091671+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"27'96","reported_seq":110,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.947441+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:11:56.947441+0000","last_peered":"2024-01-11T10:11:56.947441+0000","last_clean":"2024-01-11T10:11:56.947441+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:11:56.947441+0000","last_undegraded":"2024-01-11T10:11:56.947441+0000","last_fullsized":"2024-01-11T10:11:56.947441+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.213380+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:11:55.213380+0000","last_peered":"2024-01-11T10:11:55.213380+0000","last_clean":"2024-01-11T10:11:55.213380+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:11:55.213380+0000","last_undegraded":"2024-01-11T10:11:55.213380+0000","last_fullsized":"2024-01-11T10:11:55.213380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":78839808,"num_objects":1203,"num_object_clones":0,"num_object_copies":3609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1203,"num_write_kb":76992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1188,"ondisk_log_size":1188,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":51236,"kb_used_data":2204,"kb_used_omap":0,"kb_used_meta":49024,"kb_avail":94320604,"statfs":{"total":96636764160,"available":96584298496,"internally_reserved":0,"allocated":2256896,"data_stored":32377972,"data_compressed":1535622,"data_compressed_allocated":2023424,"data_compressed_original":32292864,"omap_allocated":0,"internal_metadata":50200576},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705879,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":55776,"kb_used_data":2584,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94316064,"statfs":{"total":96636764160,"available":96579649536,"internally_reserved":0,"allocated":2646016,"data_stored":38603892,"data_compressed":1831150,"data_compressed_allocated":2412544,"data_compressed_original":38518784,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71968,"kb_used_data":4888,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94299872,"statfs":{"total":96636764160,"available":96563068928,"internally_reserved":0,"allocated":5005312,"data_stored":75969147,"data_compressed":3604766,"data_compressed_allocated":4747264,"data_compressed_original":75874304,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67816,"kb_used_data":4896,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94304024,"statfs":{"total":96636764160,"available":96567320576,"internally_reserved":0,"allocated":5013504,"data_stored":76620718,"data_compressed":3633906,"data_compressed_allocated":4784128,"data_compressed_original":76546048,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:11:59.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 11 16 1811 1795 10.198 10.0625 0.0948981 0.0976246 2024-01-11T10:11:59.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:11:59.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:59.343+0000 7f54c4508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:11:59.332 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:11:59.348+0000 7f54c4508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:00.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 12 16 1995 1979 10.3064 11.5 0.139089 0.0965621 2024-01-11T10:12:01.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 13 16 2185 2169 10.427 11.875 0.0480135 0.0954795 2024-01-11T10:12:01.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:01.650 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:01.664+0000 7fe0de39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:01.653 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:01.669+0000 7fe0de39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:02.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 14 16 2365 2349 10.4857 11.25 0.0793376 0.0949611 2024-01-11T10:12:03.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 15 16 2550 2534 10.5575 11.5625 0.114655 0.0943752 2024-01-11T10:12:03.264 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:12:03.264 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:12:03.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:12:03.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:03.396+0000 7f4283c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:03.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:03.403+0000 7f4283c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:12:03.633 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:12:03.646 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2024-01-11T10:12:02.136592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":104923680,"num_objects":1594,"num_object_clones":0,"num_object_copies":4782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":1729,"num_write_kb":103782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1679,"ondisk_log_size":1679,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":255312,"kb_used_data":18928,"kb_used_omap":0,"kb_used_meta":236352,"kb_avail":377232048,"statfs":{"total":386547056640,"available":386285617152,"internally_reserved":0,"allocated":19382272,"data_stored":294301298,"data_compressed":13962979,"data_compressed_allocated":18386944,"data_compressed_original":293945344,"omap_allocated":0,"internal_metadata":242024448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":52035584,"num_objects":794,"num_object_clones":0,"num_object_copies":2382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":794,"num_write_kb":50816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.111746"},"pg_stats":[{"pgid":"4.f","version":"27'94","reported_seq":110,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.927379+0000","last_change":"2024-01-11T10:11:47.138050+0000","last_active":"2024-01-11T10:11:57.927379+0000","last_peered":"2024-01-11T10:11:57.927379+0000","last_clean":"2024-01-11T10:11:57.927379+0000","last_became_active":"2024-01-11T10:11:47.134738+0000","last_became_peered":"2024-01-11T10:11:47.134738+0000","last_unstale":"2024-01-11T10:11:57.927379+0000","last_undegraded":"2024-01-11T10:11:57.927379+0000","last_fullsized":"2024-01-11T10:11:57.927379+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":6080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"27'93","reported_seq":110,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.188512+0000","last_change":"2024-01-11T10:11:47.125437+0000","last_active":"2024-01-11T10:11:58.188512+0000","last_peered":"2024-01-11T10:11:58.188512+0000","last_clean":"2024-01-11T10:11:58.188512+0000","last_became_active":"2024-01-11T10:11:47.122491+0000","last_became_peered":"2024-01-11T10:11:47.122491+0000","last_unstale":"2024-01-11T10:11:58.188512+0000","last_undegraded":"2024-01-11T10:11:58.188512+0000","last_fullsized":"2024-01-11T10:11:58.188512+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6094848,"num_objects":93,"num_object_clones":0,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":93,"num_write_kb":5952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"27'102","reported_seq":116,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.037441+0000","last_change":"2024-01-11T10:11:47.235698+0000","last_active":"2024-01-11T10:11:57.037441+0000","last_peered":"2024-01-11T10:11:57.037441+0000","last_clean":"2024-01-11T10:11:57.037441+0000","last_became_active":"2024-01-11T10:11:47.229122+0000","last_became_peered":"2024-01-11T10:11:47.229122+0000","last_unstale":"2024-01-11T10:11:57.037441+0000","last_undegraded":"2024-01-11T10:11:57.037441+0000","last_fullsized":"2024-01-11T10:11:57.037441+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6750208,"num_objects":103,"num_object_clones":0,"num_object_copies":309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103,"num_write_kb":6592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"28'87","reported_seq":104,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.053448+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:11:57.053448+0000","last_peered":"2024-01-11T10:11:57.053448+0000","last_clean":"2024-01-11T10:11:57.053448+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:11:57.053448+0000","last_undegraded":"2024-01-11T10:11:57.053448+0000","last_fullsized":"2024-01-11T10:11:57.053448+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"28'96","reported_seq":113,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.207888+0000","last_change":"2024-01-11T10:11:47.214794+0000","last_active":"2024-01-11T10:11:58.207888+0000","last_peered":"2024-01-11T10:11:58.207888+0000","last_clean":"2024-01-11T10:11:58.207888+0000","last_became_active":"2024-01-11T10:11:47.211625+0000","last_became_peered":"2024-01-11T10:11:47.211625+0000","last_unstale":"2024-01-11T10:11:58.207888+0000","last_undegraded":"2024-01-11T10:11:58.207888+0000","last_fullsized":"2024-01-11T10:11:58.207888+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6291456,"num_objects":96,"num_object_clones":0,"num_object_copies":288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":96,"num_write_kb":6144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"27'91","reported_seq":105,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.983449+0000","last_change":"2024-01-11T10:11:47.281474+0000","last_active":"2024-01-11T10:11:56.983449+0000","last_peered":"2024-01-11T10:11:56.983449+0000","last_clean":"2024-01-11T10:11:56.983449+0000","last_became_active":"2024-01-11T10:11:47.278218+0000","last_became_peered":"2024-01-11T10:11:47.278218+0000","last_unstale":"2024-01-11T10:11:56.983449+0000","last_undegraded":"2024-01-11T10:11:56.983449+0000","last_fullsized":"2024-01-11T10:11:56.983449+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6029312,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92,"num_write_kb":5888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"27'101","reported_seq":115,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.072851+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:11:57.072851+0000","last_peered":"2024-01-11T10:11:57.072851+0000","last_clean":"2024-01-11T10:11:57.072851+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:11:57.072851+0000","last_undegraded":"2024-01-11T10:11:57.072851+0000","last_fullsized":"2024-01-11T10:11:57.072851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6684672,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":102,"num_write_kb":6528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"27'94","reported_seq":108,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.962228+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:11:56.962228+0000","last_peered":"2024-01-11T10:11:56.962228+0000","last_clean":"2024-01-11T10:11:56.962228+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:11:56.962228+0000","last_undegraded":"2024-01-11T10:11:56.962228+0000","last_fullsized":"2024-01-11T10:11:56.962228+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6225920,"num_objects":95,"num_object_clones":0,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":6080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"27'103","reported_seq":117,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.897844+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:11:56.897844+0000","last_peered":"2024-01-11T10:11:56.897844+0000","last_clean":"2024-01-11T10:11:56.897844+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:11:56.897844+0000","last_undegraded":"2024-01-11T10:11:56.897844+0000","last_fullsized":"2024-01-11T10:11:56.897844+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6815744,"num_objects":104,"num_object_clones":0,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":104,"num_write_kb":6656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":170,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.076470+0000","last_change":"2024-01-11T10:08:20.166269+0000","last_active":"2024-01-11T10:11:57.076470+0000","last_peered":"2024-01-11T10:11:57.076470+0000","last_clean":"2024-01-11T10:11:57.076470+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:11:57.076470+0000","last_undegraded":"2024-01-11T10:11:57.076470+0000","last_fullsized":"2024-01-11T10:11:57.076470+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.214392+0000","last_change":"2024-01-11T10:08:22.228624+0000","last_active":"2024-01-11T10:11:55.214392+0000","last_peered":"2024-01-11T10:11:55.214392+0000","last_clean":"2024-01-11T10:11:55.214392+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:11:55.214392+0000","last_undegraded":"2024-01-11T10:11:55.214392+0000","last_fullsized":"2024-01-11T10:11:55.214392+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"28'87","reported_seq":104,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.206186+0000","last_change":"2024-01-11T10:11:47.198675+0000","last_active":"2024-01-11T10:11:58.206186+0000","last_peered":"2024-01-11T10:11:58.206186+0000","last_clean":"2024-01-11T10:11:58.206186+0000","last_became_active":"2024-01-11T10:11:47.194485+0000","last_became_peered":"2024-01-11T10:11:47.194485+0000","last_unstale":"2024-01-11T10:11:58.206186+0000","last_undegraded":"2024-01-11T10:11:58.206186+0000","last_fullsized":"2024-01-11T10:11:58.206186+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5701632,"num_objects":87,"num_object_clones":0,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":87,"num_write_kb":5568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.211218+0000","last_change":"2024-01-11T10:08:22.236347+0000","last_active":"2024-01-11T10:11:58.211218+0000","last_peered":"2024-01-11T10:11:58.211218+0000","last_clean":"2024-01-11T10:11:58.211218+0000","last_became_active":"2024-01-11T10:08:22.233043+0000","last_became_peered":"2024-01-11T10:08:22.233043+0000","last_unstale":"2024-01-11T10:11:58.211218+0000","last_undegraded":"2024-01-11T10:11:58.211218+0000","last_fullsized":"2024-01-11T10:11:58.211218+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"27'114","reported_seq":130,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.917498+0000","last_change":"2024-01-11T10:11:47.157444+0000","last_active":"2024-01-11T10:11:57.917498+0000","last_peered":"2024-01-11T10:11:57.917498+0000","last_clean":"2024-01-11T10:11:57.917498+0000","last_became_active":"2024-01-11T10:11:47.154600+0000","last_became_peered":"2024-01-11T10:11:47.154600+0000","last_unstale":"2024-01-11T10:11:57.917498+0000","last_undegraded":"2024-01-11T10:11:57.917498+0000","last_fullsized":"2024-01-11T10:11:57.917498+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":115,"num_write_kb":7360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.177405+0000","last_change":"2024-01-11T10:08:22.200856+0000","last_active":"2024-01-11T10:11:55.177405+0000","last_peered":"2024-01-11T10:11:55.177405+0000","last_clean":"2024-01-11T10:11:55.177405+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:11:55.177405+0000","last_undegraded":"2024-01-11T10:11:55.177405+0000","last_fullsized":"2024-01-11T10:11:55.177405+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"27'106","reported_seq":122,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.859950+0000","last_change":"2024-01-11T10:11:47.151805+0000","last_active":"2024-01-11T10:11:57.859950+0000","last_peered":"2024-01-11T10:11:57.859950+0000","last_clean":"2024-01-11T10:11:57.859950+0000","last_became_active":"2024-01-11T10:11:47.149100+0000","last_became_peered":"2024-01-11T10:11:47.149100+0000","last_unstale":"2024-01-11T10:11:57.859950+0000","last_undegraded":"2024-01-11T10:11:57.859950+0000","last_fullsized":"2024-01-11T10:11:57.859950+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":107,"num_write_kb":6848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":33,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.198545+0000","last_change":"2024-01-11T10:08:22.237045+0000","last_active":"2024-01-11T10:11:58.198545+0000","last_peered":"2024-01-11T10:11:58.198545+0000","last_clean":"2024-01-11T10:11:58.198545+0000","last_became_active":"2024-01-11T10:08:22.232745+0000","last_became_peered":"2024-01-11T10:08:22.232745+0000","last_unstale":"2024-01-11T10:11:58.198545+0000","last_undegraded":"2024-01-11T10:11:58.198545+0000","last_fullsized":"2024-01-11T10:11:58.198545+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"27'99","reported_seq":113,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.040994+0000","last_change":"2024-01-11T10:11:47.206076+0000","last_active":"2024-01-11T10:11:57.040994+0000","last_peered":"2024-01-11T10:11:57.040994+0000","last_clean":"2024-01-11T10:11:57.040994+0000","last_became_active":"2024-01-11T10:11:47.203256+0000","last_became_peered":"2024-01-11T10:11:47.203256+0000","last_unstale":"2024-01-11T10:11:57.040994+0000","last_undegraded":"2024-01-11T10:11:57.040994+0000","last_fullsized":"2024-01-11T10:11:57.040994+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6553600,"num_objects":100,"num_object_clones":0,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":100,"num_write_kb":6400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.185103+0000","last_change":"2024-01-11T10:08:22.240331+0000","last_active":"2024-01-11T10:11:55.185103+0000","last_peered":"2024-01-11T10:11:55.185103+0000","last_clean":"2024-01-11T10:11:55.185103+0000","last_became_active":"2024-01-11T10:08:22.234869+0000","last_became_peered":"2024-01-11T10:08:22.234869+0000","last_unstale":"2024-01-11T10:11:55.185103+0000","last_undegraded":"2024-01-11T10:11:55.185103+0000","last_fullsized":"2024-01-11T10:11:55.185103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"28'118","reported_seq":135,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.213459+0000","last_change":"2024-01-11T10:11:47.209065+0000","last_active":"2024-01-11T10:11:58.213459+0000","last_peered":"2024-01-11T10:11:58.213459+0000","last_clean":"2024-01-11T10:11:58.213459+0000","last_became_active":"2024-01-11T10:11:47.196382+0000","last_became_peered":"2024-01-11T10:11:47.196382+0000","last_unstale":"2024-01-11T10:11:58.213459+0000","last_undegraded":"2024-01-11T10:11:58.213459+0000","last_fullsized":"2024-01-11T10:11:58.213459+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":118,"log_dups_size":0,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7733248,"num_objects":118,"num_object_clones":0,"num_object_copies":354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":7552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:57.077614+0000","last_change":"2024-01-11T10:08:22.224534+0000","last_active":"2024-01-11T10:11:57.077614+0000","last_peered":"2024-01-11T10:11:57.077614+0000","last_clean":"2024-01-11T10:11:57.077614+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:11:57.077614+0000","last_undegraded":"2024-01-11T10:11:57.077614+0000","last_fullsized":"2024-01-11T10:11:57.077614+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"28'101","reported_seq":118,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-01-11T10:11:58.217928+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:11:58.217928+0000","last_peered":"2024-01-11T10:11:58.217928+0000","last_clean":"2024-01-11T10:11:58.217928+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:11:58.217928+0000","last_undegraded":"2024-01-11T10:11:58.217928+0000","last_fullsized":"2024-01-11T10:11:58.217928+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":101,"num_write_kb":6464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.187751+0000","last_change":"2024-01-11T10:08:22.211506+0000","last_active":"2024-01-11T10:11:55.187751+0000","last_peered":"2024-01-11T10:11:55.187751+0000","last_clean":"2024-01-11T10:11:55.187751+0000","last_became_active":"2024-01-11T10:08:22.208367+0000","last_became_peered":"2024-01-11T10:08:22.208367+0000","last_unstale":"2024-01-11T10:11:55.187751+0000","last_undegraded":"2024-01-11T10:11:55.187751+0000","last_fullsized":"2024-01-11T10:11:55.187751+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1","version":"27'96","reported_seq":110,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:56.947441+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:11:56.947441+0000","last_peered":"2024-01-11T10:11:56.947441+0000","last_clean":"2024-01-11T10:11:56.947441+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:11:56.947441+0000","last_undegraded":"2024-01-11T10:11:56.947441+0000","last_fullsized":"2024-01-11T10:11:56.947441+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6356992,"num_objects":97,"num_object_clones":0,"num_object_copies":291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":97,"num_write_kb":6208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":31,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-01-11T10:11:55.213380+0000","last_change":"2024-01-11T10:08:22.219863+0000","last_active":"2024-01-11T10:11:55.213380+0000","last_peered":"2024-01-11T10:11:55.213380+0000","last_clean":"2024-01-11T10:11:55.213380+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:11:55.213380+0000","last_undegraded":"2024-01-11T10:11:55.213380+0000","last_fullsized":"2024-01-11T10:11:55.213380+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":104333312,"num_objects":1592,"num_object_clones":0,"num_object_copies":4776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1592,"num_write_kb":101888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1582,"ondisk_log_size":1582,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":57348,"kb_used_data":4156,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94314492,"statfs":{"total":96636764160,"available":96578039808,"internally_reserved":0,"allocated":4255744,"data_stored":64103886,"data_compressed":3041683,"data_compressed_allocated":4005888,"data_compressed_original":64012288,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58180,"kb_used_data":4988,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":94313660,"statfs":{"total":96636764160,"available":96577187840,"internally_reserved":0,"allocated":5107712,"data_stored":77607547,"data_compressed":3682624,"data_compressed_allocated":4849664,"data_compressed_original":77512704,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":71968,"kb_used_data":4888,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94299872,"statfs":{"total":96636764160,"available":96563068928,"internally_reserved":0,"allocated":5005312,"data_stored":75969147,"data_compressed":3604766,"data_compressed_allocated":4747264,"data_compressed_original":75874304,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705883,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":67816,"kb_used_data":4896,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94304024,"statfs":{"total":96636764160,"available":96567320576,"internally_reserved":0,"allocated":5013504,"data_stored":76620718,"data_compressed":3633906,"data_compressed_allocated":4784128,"data_compressed_original":76546048,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:12:03.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:03.999 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:04.014+0000 7f789d79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:04.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:04.019+0000 7f789d79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:04.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 16 16 2710 2694 10.5226 10 0.120136 0.0945548 2024-01-11T10:12:05.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 17 16 2893 2877 10.5763 11.4375 0.0156882 0.0942446 2024-01-11T10:12:06.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 18 16 3064 3048 10.5825 10.6875 0.0191692 0.0941336 2024-01-11T10:12:06.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:06.378 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:06.393+0000 7f4020ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:06.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:06.397+0000 7f4020ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:07.057 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 19 16 3241 3225 10.6077 11.0625 0.152886 0.0939587 2024-01-11T10:12:08.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:12:08.075584+0000 min lat: 0.0101561 max lat: 0.312611 avg lat: 0.0941544 2024-01-11T10:12:08.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:12:08.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 20 16 3403 3387 10.5835 10.125 0.179231 0.0941544 2024-01-11T10:12:08.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:08.648 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:12:08.648 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:12:08.648 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:12:08.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:12:08.674 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:08.689+0000 7f123f35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:08.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:08.694+0000 7f123f35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:08.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:08.783+0000 7fc9c755e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:08.775 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:08.791+0000 7fc9c755e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:08.802 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:08.802 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:08.802 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:08.802 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:08.802 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:08.803 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:08.806 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:08.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:12:09.036 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:09.047+0000 7f4c2f417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:09.040 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:09.056+0000 7f4c2f417640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:09.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 21 16 3555 3539 10.5319 9.5 0.191763 0.0945404 2024-01-11T10:12:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:09.106 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:09.107 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:09.110 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:09.110 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:12:09.232 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:09.243+0000 7f74dd55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:09.236 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:09.252+0000 7f74dd55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:09.251 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:09.252 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:09.255 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:09.255 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:12:09.380 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:09.392+0000 7fcc63f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:09.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:09.400+0000 7fcc63f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:09.411 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:09.412 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:09.412 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:09.412 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:09.412 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:09.412 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:09.414 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:10.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 22 16 3740 3724 10.5787 11.5625 0.110672 0.0943052 2024-01-11T10:12:10.844 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:10.959 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:10.974+0000 7ff0e0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:10.962 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:10.978+0000 7ff0e0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:11.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 23 16 3913 3897 10.5888 10.8125 0.177207 0.0941902 2024-01-11T10:12:12.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 24 16 4083 4067 10.5903 10.625 0.0431667 0.094229 2024-01-11T10:12:13.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 25 16 4254 4238 10.5941 10.6875 0.0390964 0.0942428 2024-01-11T10:12:13.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:13.298 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:13.313+0000 7fa0c6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:13.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:13.318+0000 7fa0c6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 26 16 4427 4411 10.6025 10.8125 0.10413 0.094084 2024-01-11T10:12:14.415 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:12:14.416 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:12:14.416 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:12:14.416 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:12:14.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:14.548+0000 7f43f16de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.541 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:14.558+0000 7f43f16de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.563 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:14.563 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:14.566 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:14.566 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:12:14.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:14.702+0000 7fa0eff55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.694 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:14.710+0000 7fa0eff55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.779 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:14.780 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:14.781 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:14.781 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:14.782 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:14.783 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:12:14.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:14.931+0000 7faabfeeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.924 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:14.941+0000 7faabfeeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:14.939 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:14.939 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:14.939 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:14.939 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:14.939 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:14.940 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:14.942 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:14.942 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:14.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:12:15.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 27 16 4579 4563 10.5616 9.5 0.0215106 0.094478 2024-01-11T10:12:15.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:15.072+0000 7fda9fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:15.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:15.081+0000 7fda9fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:15.082 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:15.083 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:15.084 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:15.084 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:15.084 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:15.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:15.086 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:15.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:15.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:15.622+0000 7f5d67820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:15.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:15.628+0000 7f5d67820640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:16.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 28 16 4747 4731 10.5594 10.5 0.1501 0.0944061 2024-01-11T10:12:17.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 29 16 4921 4905 10.5703 10.875 0.164518 0.094384 2024-01-11T10:12:17.802 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:17.907 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:17.922+0000 7f40c415e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:17.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:17.926+0000 7f40c415e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:18.058 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 30 16 5046 5030 10.4783 7.8125 0.0120882 0.0940936 2024-01-11T10:12:19.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 31 16 5046 5030 10.1403 0 - 0.0940936 2024-01-11T10:12:19.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:20.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 32 16 5046 5030 9.82337 0 - 0.0940936 2024-01-11T10:12:20.087 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:12:20.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:12:20.088 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:12:20.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:12:20.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.120+0000 7f791891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.108 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.125+0000 7f791891e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.211+0000 7fcdbab04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.218+0000 7fcdbab04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:20.215 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:20.216 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:20.218 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:20.218 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:20.218 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:12:20.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.379+0000 7f972f8f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.370 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.386+0000 7f972f8f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:20.383 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:20.384 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:20.384 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:20.384 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:20.384 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:20.384 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:20.387 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:20.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:12:20.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.506+0000 7efc82c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:20.497 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:20.513+0000 7efc82c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:21.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 33 16 5046 5030 9.52568 0 - 0.0940936 2024-01-11T10:12:22.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 34 16 5046 5030 9.24551 0 - 0.0940936 2024-01-11T10:12:23.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 35 16 5046 5030 8.98133 0 - 0.0940936 2024-01-11T10:12:23.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-11T10:12:24.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 36 16 5046 5030 8.73183 0 - 0.0940936 2024-01-11T10:12:25.059 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 37 16 5046 5030 8.49582 0 - 0.0940936 2024-01-11T10:12:26.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 38 16 5046 5030 8.27223 0 - 0.0940936 2024-01-11T10:12:27.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 39 16 5046 5030 8.06011 0 - 0.0940936 2024-01-11T10:12:28.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:12:28.077878+0000 min lat: 0.010024 max lat: 0.312611 avg lat: 0.0940936 2024-01-11T10:12:28.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:12:28.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 40 16 5046 5030 7.8586 0 - 0.0940936 2024-01-11T10:12:29.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 41 16 5046 5030 7.66692 0 - 0.0940936 2024-01-11T10:12:29.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~6s 2024-01-11T10:12:30.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 42 16 5046 5030 7.48436 0 - 0.0940936 2024-01-11T10:12:31.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 43 16 5046 5030 7.3103 0 - 0.0940936 2024-01-11T10:12:32.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 44 16 5046 5030 7.14415 0 - 0.0940936 2024-01-11T10:12:33.060 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 45 16 5046 5030 6.98538 0 - 0.0940936 2024-01-11T10:12:34.061 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 46 16 5046 5030 6.83352 0 - 0.0940936 2024-01-11T10:12:34.650 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~11s 2024-01-11T10:12:35.061 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 47 16 5046 5030 6.68812 0 - 0.0940936 2024-01-11T10:12:36.061 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 48 16 5046 5030 6.54878 0 - 0.0940936 2024-01-11T10:12:37.061 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 49 16 5046 5030 6.41512 0 - 0.0940936 2024-01-11T10:12:38.061 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 50 16 5046 5030 6.28681 0 - 0.0940936 2024-01-11T10:12:39.061 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 51 16 5046 5030 6.16353 0 - 0.0940936 2024-01-11T10:12:40.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 52 16 5046 5030 6.045 0 - 0.0940936 2024-01-11T10:12:40.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~17s 2024-01-11T10:12:41.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 53 16 5046 5030 5.93094 0 - 0.0940936 2024-01-11T10:12:42.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 54 16 5046 5030 5.8211 0 - 0.0940936 2024-01-11T10:12:43.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 55 16 5046 5030 5.71526 0 - 0.0940936 2024-01-11T10:12:44.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 56 16 5046 5030 5.6132 0 - 0.0940936 2024-01-11T10:12:45.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 57 16 5046 5030 5.51472 0 - 0.0940936 2024-01-11T10:12:45.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~22s 2024-01-11T10:12:46.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 58 16 5046 5030 5.41963 0 - 0.0940936 2024-01-11T10:12:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:47.001 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:47.005 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:47.005 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:12:47.062 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 59 16 5046 5030 5.32777 0 - 0.0940936 2024-01-11T10:12:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:47.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:12:47.138 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:47.149+0000 7f21a24f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:47.142 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:47.159+0000 7f21a24f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:47.158 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:47.159 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:47.161 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:48.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:12:48.080860+0000 min lat: 0.010024 max lat: 0.312611 avg lat: 0.0940936 2024-01-11T10:12:48.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:12:48.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 60 16 5046 5030 5.23898 0 - 0.0940936 2024-01-11T10:12:49.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 61 16 5202 5186 5.31291 0.314516 0.0461425 0.188087 2024-01-11T10:12:49.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:49.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:49.209+0000 7f0dbf59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:49.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:49.214+0000 7f0dbf59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:49.329 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:49.348 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:12:50.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 62 16 5376 5360 5.40261 10.875 0.103234 0.184944 2024-01-11T10:12:51.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 63 16 5549 5533 5.48846 10.8125 0.121341 0.181989 2024-01-11T10:12:51.164 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~28s 2024-01-11T10:12:51.349 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:51.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:51.483+0000 7fd817d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:51.471 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:51.488+0000 7fd817d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:51.576 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:51.598 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:12:52.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 64 16 5724 5708 5.57359 10.9375 0.0811301 0.179273 2024-01-11T10:12:52.162 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:12:52.162 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:12:52.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:12:52.281 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:52.294+0000 7f249d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:52.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:52.302+0000 7f249d4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:52.548 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:12:52.548 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:12:52.560 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-01-11T10:12:52.146812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247857696,"num_objects":3775,"num_object_clones":0,"num_object_copies":11325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3910,"num_write_kb":243366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5140,"ondisk_log_size":5140,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":342404,"kb_used_data":61604,"kb_used_omap":0,"kb_used_meta":280768,"kb_avail":377144956,"statfs":{"total":386547056640,"available":386196434944,"internally_reserved":0,"allocated":63082496,"data_stored":992610492,"data_compressed":47121339,"data_compressed_allocated":62029824,"data_compressed_original":992231424,"omap_allocated":0,"internal_metadata":287506432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-83230720,"num_objects":-1270,"num_object_clones":0,"num_object_copies":-3810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1270,"num_write_kb":-81280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001835"},"pg_stats":[{"pgid":"4.f","version":"29'320","reported_seq":3,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:47.021102+0000","last_change":"2024-01-11T10:12:47.064648+0000","last_active":"2024-01-11T10:11:45.990370+0000","last_peered":"2024-01-11T10:11:45.990370+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T10:11:47.021102+0000","last_undegraded":"2024-01-11T10:11:47.021102+0000","last_fullsized":"2024-01-11T10:11:47.021102+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"29'306","reported_seq":3,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:47.032339+0000","last_change":"2024-01-11T10:12:47.074142+0000","last_active":"2024-01-11T10:11:45.990370+0000","last_peered":"2024-01-11T10:11:45.990370+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T10:11:47.032339+0000","last_undegraded":"2024-01-11T10:11:47.032339+0000","last_fullsized":"2024-01-11T10:11:47.032339+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"29'304","reported_seq":322,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:12:17.483462+0000","last_change":"2024-01-11T10:12:47.066594+0000","last_active":"2024-01-11T10:12:47.066594+0000","last_peered":"2024-01-11T10:12:17.483462+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:11:47.229122+0000","last_became_peered":"2024-01-11T10:11:47.229122+0000","last_unstale":"2024-01-11T10:12:17.483462+0000","last_undegraded":"2024-01-11T10:12:17.483462+0000","last_fullsized":"2024-01-11T10:12:17.483462+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19922944,"num_objects":304,"num_object_clones":0,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":304,"num_write_kb":19456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"29'302","reported_seq":321,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-11T10:12:47.069419+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:12:47.069419+0000","last_peered":"2024-01-11T10:12:47.069419+0000","last_clean":"2024-01-11T10:12:47.069419+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:12:47.069419+0000","last_undegraded":"2024-01-11T10:12:47.069419+0000","last_fullsized":"2024-01-11T10:12:47.069419+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19791872,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":302,"num_write_kb":19328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"29'325","reported_seq":349,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:48.198609+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:12:48.198609+0000","last_peered":"2024-01-11T10:12:48.198609+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:12:48.198609+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":325,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":325}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"29'326","reported_seq":344,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:12:17.672029+0000","last_change":"2024-01-11T10:12:47.134563+0000","last_active":"2024-01-11T10:12:47.134563+0000","last_peered":"2024-01-11T10:12:17.672029+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:11:47.278218+0000","last_became_peered":"2024-01-11T10:11:47.278218+0000","last_unstale":"2024-01-11T10:12:17.672029+0000","last_undegraded":"2024-01-11T10:12:17.672029+0000","last_fullsized":"2024-01-11T10:12:17.672029+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":326,"num_object_clones":0,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":326,"num_write_kb":20864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"29'325","reported_seq":344,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-11T10:12:47.079562+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:12:47.079562+0000","last_peered":"2024-01-11T10:12:47.079562+0000","last_clean":"2024-01-11T10:12:47.079562+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:12:47.079562+0000","last_undegraded":"2024-01-11T10:12:47.079562+0000","last_fullsized":"2024-01-11T10:12:47.079562+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":325,"num_write_kb":20800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"29'319","reported_seq":338,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-11T10:12:47.116384+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:12:47.116384+0000","last_peered":"2024-01-11T10:12:47.116384+0000","last_clean":"2024-01-11T10:12:47.116384+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:12:47.116384+0000","last_undegraded":"2024-01-11T10:12:47.116384+0000","last_fullsized":"2024-01-11T10:12:47.116384+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20905984,"num_objects":319,"num_object_clones":0,"num_object_copies":957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":319,"num_write_kb":20416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"29'335","reported_seq":354,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-11T10:12:47.074977+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:12:47.074977+0000","last_peered":"2024-01-11T10:12:47.074977+0000","last_clean":"2024-01-11T10:12:47.074977+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:12:47.074977+0000","last_undegraded":"2024-01-11T10:12:47.074977+0000","last_fullsized":"2024-01-11T10:12:47.074977+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21954560,"num_objects":335,"num_object_clones":0,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":335,"num_write_kb":21440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":171,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:57.077128+0000","last_change":"2024-01-11T10:12:47.041206+0000","last_active":"2024-01-11T10:12:47.041206+0000","last_peered":"2024-01-11T10:11:57.077128+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:08:20.157177+0000","last_became_peered":"2024-01-11T10:08:20.157177+0000","last_unstale":"2024-01-11T10:11:57.077128+0000","last_undegraded":"2024-01-11T10:11:57.077128+0000","last_fullsized":"2024-01-11T10:11:57.077128+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:58.304116+0000","last_change":"2024-01-11T10:12:47.133327+0000","last_active":"2024-01-11T10:12:47.133327+0000","last_peered":"2024-01-11T10:11:58.304116+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:08:22.224730+0000","last_became_peered":"2024-01-11T10:08:22.224730+0000","last_unstale":"2024-01-11T10:11:58.304116+0000","last_undegraded":"2024-01-11T10:11:58.304116+0000","last_fullsized":"2024-01-11T10:11:58.304116+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"29'308","reported_seq":332,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:48.155997+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:12:48.155997+0000","last_peered":"2024-01-11T10:12:48.155997+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:12:48.155997+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185088,"num_objects":308,"num_object_clones":0,"num_object_copies":924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":308,"num_write_kb":19712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":308}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.158071+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:12:48.158071+0000","last_peered":"2024-01-11T10:12:48.158071+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:12:48.158071+0000","last_undegraded":"2024-01-11T10:12:48.158071+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"29'332","reported_seq":10,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.144714+0000","last_change":"2024-01-11T10:12:48.140341+0000","last_active":"2024-01-11T10:12:48.144714+0000","last_peered":"2024-01-11T10:12:48.144714+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:12:48.144714+0000","last_undegraded":"2024-01-11T10:12:48.144714+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:58.293562+0000","last_change":"2024-01-11T10:12:47.106485+0000","last_active":"2024-01-11T10:12:47.106485+0000","last_peered":"2024-01-11T10:11:58.293562+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:08:22.194671+0000","last_became_peered":"2024-01-11T10:08:22.194671+0000","last_unstale":"2024-01-11T10:11:58.293562+0000","last_undegraded":"2024-01-11T10:11:58.293562+0000","last_fullsized":"2024-01-11T10:11:58.293562+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"29'314","reported_seq":3,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:47.061306+0000","last_change":"2024-01-11T10:12:47.039973+0000","last_active":"2024-01-11T10:11:45.990370+0000","last_peered":"2024-01-11T10:11:45.990370+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T10:11:47.061306+0000","last_undegraded":"2024-01-11T10:11:47.061306+0000","last_fullsized":"2024-01-11T10:11:47.061306+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.174832+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:12:48.174832+0000","last_peered":"2024-01-11T10:12:48.174832+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:12:48.174832+0000","last_undegraded":"2024-01-11T10:12:48.174832+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"29'303","reported_seq":321,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:12:17.648540+0000","last_change":"2024-01-11T10:12:47.114784+0000","last_active":"2024-01-11T10:12:47.114784+0000","last_peered":"2024-01-11T10:12:17.648540+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:11:47.203256+0000","last_became_peered":"2024-01-11T10:11:47.203256+0000","last_unstale":"2024-01-11T10:12:17.648540+0000","last_undegraded":"2024-01-11T10:12:17.648540+0000","last_fullsized":"2024-01-11T10:12:17.648540+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":303,"num_write_kb":19392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":3,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:08:22.139533+0000","last_change":"2024-01-11T10:12:47.038226+0000","last_active":"2024-01-11T10:08:21.115873+0000","last_peered":"2024-01-11T10:08:21.115873+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-11T10:08:22.139533+0000","last_undegraded":"2024-01-11T10:08:22.139533+0000","last_fullsized":"2024-01-11T10:08:22.139533+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"29'316","reported_seq":341,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:48.168773+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:12:48.168773+0000","last_peered":"2024-01-11T10:12:48.168773+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:12:48.168773+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20774912,"num_objects":317,"num_object_clones":0,"num_object_copies":951,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":317,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":317,"num_write_kb":20288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":317}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:57.077768+0000","last_change":"2024-01-11T10:12:47.044561+0000","last_active":"2024-01-11T10:12:47.044561+0000","last_peered":"2024-01-11T10:11:57.077768+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:08:22.220671+0000","last_became_peered":"2024-01-11T10:08:22.220671+0000","last_unstale":"2024-01-11T10:11:57.077768+0000","last_undegraded":"2024-01-11T10:11:57.077768+0000","last_fullsized":"2024-01-11T10:11:57.077768+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"29'296","reported_seq":317,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:48.071156+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:12:48.071156+0000","last_peered":"2024-01-11T10:12:48.071156+0000","last_clean":"2024-01-11T10:12:48.071156+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:12:48.071156+0000","last_undegraded":"2024-01-11T10:12:48.071156+0000","last_fullsized":"2024-01-11T10:12:48.071156+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398656,"num_objects":296,"num_object_clones":0,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":296,"num_write_kb":18944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161356+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:12:48.161356+0000","last_peered":"2024-01-11T10:12:48.161356+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:12:48.161356+0000","last_undegraded":"2024-01-11T10:12:48.161356+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"29'312","reported_seq":331,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-01-11T10:12:47.087596+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:12:47.087596+0000","last_peered":"2024-01-11T10:12:47.087596+0000","last_clean":"2024-01-11T10:12:47.087596+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:12:47.087596+0000","last_undegraded":"2024-01-11T10:12:47.087596+0000","last_fullsized":"2024-01-11T10:12:47.087596+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":312,"num_write_kb":19968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":36,"reported_epoch":30,"state":"peering","last_fresh":"2024-01-11T10:11:58.305690+0000","last_change":"2024-01-11T10:12:47.084097+0000","last_active":"2024-01-11T10:12:47.084097+0000","last_peered":"2024-01-11T10:11:58.305690+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:08:22.215494+0000","last_became_peered":"2024-01-11T10:08:22.215494+0000","last_unstale":"2024-01-11T10:11:58.305690+0000","last_undegraded":"2024-01-11T10:11:58.305690+0000","last_fullsized":"2024-01-11T10:11:58.305690+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":247267328,"num_objects":3773,"num_object_clones":0,"num_object_copies":11319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":951,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3773,"num_write_kb":241472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5043,"ondisk_log_size":5043,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":82564,"kb_used_data":15484,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94289276,"statfs":{"total":96636764160,"available":96552218624,"internally_reserved":0,"allocated":15855616,"data_stored":249318756,"data_compressed":11836064,"data_compressed_allocated":15581184,"data_compressed_original":249217024,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92092,"kb_used_data":16692,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94279748,"statfs":{"total":96636764160,"available":96542461952,"internally_reserved":0,"allocated":17092608,"data_stored":269238056,"data_compressed":12782054,"data_compressed_allocated":16826368,"data_compressed_original":269139968,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705933,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91972,"kb_used_data":16572,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94279868,"statfs":{"total":96636764160,"available":96542584832,"internally_reserved":0,"allocated":16969728,"data_stored":267664648,"data_compressed":12705386,"data_compressed_allocated":16723968,"data_compressed_original":267583488,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:12:52.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 4.b 4.6 2024-01-11T10:12:52.685 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:52.697+0000 7f2257b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:52.689 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:52.706+0000 7f2257b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:52.933 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.6,4.b] on osd.0 to force-recovery; 2024-01-11T10:12:52.933 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:12:53.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 65 16 5897 5881 5.65417 10.8125 0.109684 0.176694 2024-01-11T10:12:53.600 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:53.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:53.747+0000 7f9526d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:53.735 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:53.753+0000 7f9526d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:53.899 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:53.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:12:54.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 66 16 6071 6055 5.73326 10.875 0.0113316 0.174243 2024-01-11T10:12:55.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 67 16 6263 6247 5.82677 12 0.119044 0.171497 2024-01-11T10:12:55.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:12:56.043 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:56.059+0000 7fbac8ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:56.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:56.065+0000 7fbac8ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:56.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 68 16 6439 6423 5.90283 11 0.0844719 0.169305 2024-01-11T10:12:56.250 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:56.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:12:56.668 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~33s 2024-01-11T10:12:57.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 69 16 6617 6601 5.9785 11.125 0.0139293 0.167139 2024-01-11T10:12:57.947 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:12:57.948 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:12:57.948 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:12:57.948 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:12:58.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 70 16 6783 6767 6.04129 10.375 0.173916 0.165302 2024-01-11T10:12:58.064 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.076+0000 7f5b9a4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.068 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.085+0000 7f5b9a4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.092 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:58.093 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:58.096 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:58.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:12:58.221 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.233+0000 7f15ddd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.224 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.242+0000 7f15ddd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:58.313 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:58.314 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:58.314 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:58.314 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:58.314 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:58.314 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:58.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:58.317 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:58.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:12:58.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.408+0000 7fb2e235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.395 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.413+0000 7fb2e235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.495+0000 7f3beae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.488 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.506+0000 7f3beae6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.492 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:58.496 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:58.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:12:58.537 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:12:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:12:58.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:12:58.655 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.667+0000 7f4773e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:12:58.676+0000 7f4773e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:12:58.681 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:12:58.682 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:12:58.682 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:12:58.682 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:12:58.682 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:12:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:12:58.684 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:12:59.063 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 71 16 6942 6926 6.09616 9.9375 0.108994 0.163916 2024-01-11T10:13:00.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 72 16 7130 7114 6.17467 11.75 0.0515494 0.161833 2024-01-11T10:13:00.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:00.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:00.691+0000 7f89e559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:00.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:00.696+0000 7f89e559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:00.764 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:00.784 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:00.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:01.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 73 16 7308 7292 6.24247 11.125 0.0594323 0.160083 2024-01-11T10:13:02.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 74 16 7478 7462 6.30167 10.625 0.0775367 0.158567 2024-01-11T10:13:02.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~39s 2024-01-11T10:13:02.785 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:02.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:02.934+0000 7fa37d62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:02.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:02.939+0000 7fa37d62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:03.056 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:03.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 75 16 7645 7629 6.3568 10.4375 0.131068 0.157175 2024-01-11T10:13:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:03.080 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:03.685 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:03.685 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:03.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:03.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:03.837+0000 7f2cd2e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:03.827 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:03.845+0000 7f2cd2e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:04.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 76 16 7829 7813 6.42446 11.5 0.0536604 0.15558 2024-01-11T10:13:04.078 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:04.078 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:04.093 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2024-01-11T10:13:02.148718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":355664416,"num_objects":5420,"num_object_clones":0,"num_object_copies":16260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":5555,"num_write_kb":348646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6771,"ondisk_log_size":6771,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":340548,"kb_used_data":76388,"kb_used_omap":0,"kb_used_meta":264128,"kb_avail":377146812,"statfs":{"total":386547056640,"available":386198335488,"internally_reserved":0,"allocated":78221312,"data_stored":1234193259,"data_compressed":58592171,"data_compressed_allocated":77127680,"data_compressed_original":1233797120,"omap_allocated":0,"internal_metadata":270467072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1954,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1645,"num_write_kb":105280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002460"},"pg_stats":[{"pgid":"4.f","version":"31'425","reported_seq":115,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.120618+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:12:57.120618+0000","last_peered":"2024-01-11T10:12:57.120618+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:12:57.120618+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":106,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":106,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106,"num_write_kb":6784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":106}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"31'413","reported_seq":117,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.039263+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:12:57.039263+0000","last_peered":"2024-01-11T10:12:57.039263+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:12:57.039263+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7077888,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":108,"num_write_kb":6912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":108}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"31'403","reported_seq":428,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:56.604784+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:12:56.604784+0000","last_peered":"2024-01-11T10:12:56.604784+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:12:56.604784+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":404}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'402","reported_seq":424,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:57.060525+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:12:57.060525+0000","last_peered":"2024-01-11T10:12:57.060525+0000","last_clean":"2024-01-11T10:12:57.060525+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:12:57.060525+0000","last_undegraded":"2024-01-11T10:12:57.060525+0000","last_fullsized":"2024-01-11T10:12:57.060525+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":403,"num_write_kb":25792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"31'423","reported_seq":448,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.916845+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:12:57.916845+0000","last_peered":"2024-01-11T10:12:57.916845+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:12:57.916845+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":423,"log_dups_size":0,"ondisk_log_size":423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27787264,"num_objects":424,"num_object_clones":0,"num_object_copies":1272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":424,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":424,"num_write_kb":27136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":424}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"31'417","reported_seq":442,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.155035+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:12:57.155035+0000","last_peered":"2024-01-11T10:12:57.155035+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:12:57.155035+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":418,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"31'429","reported_seq":451,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:57.085685+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:12:57.085685+0000","last_peered":"2024-01-11T10:12:57.085685+0000","last_clean":"2024-01-11T10:12:57.085685+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:12:57.085685+0000","last_undegraded":"2024-01-11T10:12:57.085685+0000","last_fullsized":"2024-01-11T10:12:57.085685+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":430,"num_object_clones":0,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":430,"num_write_kb":27520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"31'424","reported_seq":446,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:57.098089+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:12:57.098089+0000","last_peered":"2024-01-11T10:12:57.098089+0000","last_clean":"2024-01-11T10:12:57.098089+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:12:57.098089+0000","last_undegraded":"2024-01-11T10:12:57.098089+0000","last_fullsized":"2024-01-11T10:12:57.098089+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27852800,"num_objects":425,"num_object_clones":0,"num_object_copies":1275,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":27200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'436","reported_seq":458,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:57.153058+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:12:57.153058+0000","last_peered":"2024-01-11T10:12:57.153058+0000","last_clean":"2024-01-11T10:12:57.153058+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:12:57.153058+0000","last_undegraded":"2024-01-11T10:12:57.153058+0000","last_fullsized":"2024-01-11T10:12:57.153058+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28639232,"num_objects":437,"num_object_clones":0,"num_object_copies":1311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":437,"num_write_kb":27968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":177,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:48.134123+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:12:48.134123+0000","last_peered":"2024-01-11T10:12:48.134123+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:12:48.134123+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.196395+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:12:48.196395+0000","last_peered":"2024-01-11T10:12:48.196395+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:12:48.196395+0000","last_undegraded":"2024-01-11T10:12:48.196395+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"31'422","reported_seq":447,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:58.210080+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:12:58.210080+0000","last_peered":"2024-01-11T10:12:58.210080+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:12:58.210080+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27721728,"num_objects":423,"num_object_clones":0,"num_object_copies":1269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":423,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":423,"num_write_kb":27072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":423}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.158071+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:12:48.158071+0000","last_peered":"2024-01-11T10:12:48.158071+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:12:48.158071+0000","last_undegraded":"2024-01-11T10:12:48.158071+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"31'444","reported_seq":122,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.904257+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:12:57.904257+0000","last_peered":"2024-01-11T10:12:57.904257+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:12:57.904257+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113,"num_write_kb":7232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.142679+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:12:48.142679+0000","last_peered":"2024-01-11T10:12:48.142679+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:12:48.142679+0000","last_undegraded":"2024-01-11T10:12:48.142679+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"31'394","reported_seq":90,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.012632+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:12:57.012632+0000","last_peered":"2024-01-11T10:12:57.012632+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:12:57.012632+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5308416,"num_objects":81,"num_object_clones":0,"num_object_copies":243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81,"num_write_kb":5184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":81}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.174832+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:12:48.174832+0000","last_peered":"2024-01-11T10:12:48.174832+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:12:48.174832+0000","last_undegraded":"2024-01-11T10:12:48.174832+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"31'406","reported_seq":431,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:56.989718+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:12:56.989718+0000","last_peered":"2024-01-11T10:12:56.989718+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:12:56.989718+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26673152,"num_objects":407,"num_object_clones":0,"num_object_copies":1221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":407,"num_write_kb":26048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":407}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.119383+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:12:48.119383+0000","last_peered":"2024-01-11T10:12:48.119383+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:12:48.119383+0000","last_undegraded":"2024-01-11T10:12:48.119383+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'418","reported_seq":443,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:57.942216+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:12:57.942216+0000","last_peered":"2024-01-11T10:12:57.942216+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:12:57.942216+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":419,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":419,"num_write_kb":26816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":419}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.118097+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:12:48.118097+0000","last_peered":"2024-01-11T10:12:48.118097+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:12:48.118097+0000","last_undegraded":"2024-01-11T10:12:48.118097+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'403","reported_seq":425,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:57.613299+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:12:57.613299+0000","last_peered":"2024-01-11T10:12:57.613299+0000","last_clean":"2024-01-11T10:12:57.613299+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:12:57.613299+0000","last_undegraded":"2024-01-11T10:12:57.613299+0000","last_fullsized":"2024-01-11T10:12:57.613299+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161356+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:12:48.161356+0000","last_peered":"2024-01-11T10:12:48.161356+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:12:48.161356+0000","last_undegraded":"2024-01-11T10:12:48.161356+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"31'415","reported_seq":437,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:12:57.118857+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:12:57.118857+0000","last_peered":"2024-01-11T10:12:57.118857+0000","last_clean":"2024-01-11T10:12:57.118857+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:12:57.118857+0000","last_undegraded":"2024-01-11T10:12:57.118857+0000","last_fullsized":"2024-01-11T10:12:57.118857+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161069+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:12:48.161069+0000","last_peered":"2024-01-11T10:12:48.161069+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:12:48.161069+0000","last_undegraded":"2024-01-11T10:12:48.161069+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":355074048,"num_objects":5418,"num_object_clones":0,"num_object_copies":16254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2903,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5418,"num_write_kb":346752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6674,"ondisk_log_size":6674,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91768,"kb_used_data":20528,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94280072,"statfs":{"total":96636764160,"available":96542793728,"internally_reserved":0,"allocated":21020672,"data_stored":331831841,"data_compressed":15754072,"data_compressed_allocated":20738048,"data_compressed_original":331726848,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":88692,"kb_used_data":21612,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94283148,"statfs":{"total":96636764160,"available":96545943552,"internally_reserved":0,"allocated":22130688,"data_stored":349592097,"data_compressed":16597366,"data_compressed_allocated":21848064,"data_compressed_original":349487104,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705943,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84312,"kb_used_data":21392,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94287528,"statfs":{"total":96636764160,"available":96550428672,"internally_reserved":0,"allocated":21905408,"data_stored":346380289,"data_compressed":16442898,"data_compressed_allocated":21643264,"data_compressed_original":346292224,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:05.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 77 16 7982 7966 6.46521 9.5625 0.136992 0.154505 2024-01-11T10:13:05.081 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:05.195 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:05.210+0000 7f9b51f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:05.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:05.215+0000 7f9b51f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:05.320 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:05.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:06.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 78 16 8144 8128 6.51212 10.125 0.0812383 0.153396 2024-01-11T10:13:07.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 79 16 8329 8313 6.57603 11.5625 0.155396 0.151964 2024-01-11T10:13:07.339 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:07.455 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:07.470+0000 7f6b5519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:07.458 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:07.475+0000 7f6b5519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:07.570 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:07.592 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:07.592 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:07.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~44s 2024-01-11T10:13:08.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:13:08.082532+0000 min lat: 0.00955844 max lat: 30.6859 avg lat: 0.150793 2024-01-11T10:13:08.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:13:08.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 80 15 8498 8483 6.62663 10.625 0.0453807 0.150793 2024-01-11T10:13:09.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 81 16 8697 8681 6.69758 12.375 0.0716982 0.149221 2024-01-11T10:13:09.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:09.096 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:09.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:09.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:09.252+0000 7f1e1959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:09.242 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:09.259+0000 7f1e1959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:09.454 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:09.467 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-01-11T10:13:08.149810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":450363936,"num_objects":6865,"num_object_clones":0,"num_object_copies":20595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7000,"num_write_kb":441126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8216,"ondisk_log_size":8216,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":384876,"kb_used_data":90188,"kb_used_omap":0,"kb_used_meta":294656,"kb_avail":377102484,"statfs":{"total":386547056640,"available":386152943616,"internally_reserved":0,"allocated":92352512,"data_stored":1460292459,"data_compressed":69328571,"data_compressed_allocated":91258880,"data_compressed_original":1459896320,"omap_allocated":0,"internal_metadata":301727744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":152567808,"num_objects":2328,"num_object_clones":0,"num_object_copies":6984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1361,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2328,"num_write_kb":148992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002255"},"pg_stats":[{"pgid":"4.f","version":"31'538","reported_seq":228,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:07.174100+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:07.174100+0000","last_peered":"2024-01-11T10:13:07.174100+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:07.174100+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":538,"log_dups_size":0,"ondisk_log_size":538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14352384,"num_objects":219,"num_object_clones":0,"num_object_copies":657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":219,"num_write_kb":14016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":219}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"31'517","reported_seq":221,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:07.047623+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:07.047623+0000","last_peered":"2024-01-11T10:13:07.047623+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:07.047623+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":212,"num_write_kb":13568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":212}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"31'502","reported_seq":527,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:07.094443+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:07.094443+0000","last_peered":"2024-01-11T10:13:07.094443+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:07.094443+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":502,"log_dups_size":0,"ondisk_log_size":502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32964608,"num_objects":503,"num_object_clones":0,"num_object_copies":1509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":503,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":503,"num_write_kb":32192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":503}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"31'513","reported_seq":535,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:07.154347+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:07.154347+0000","last_peered":"2024-01-11T10:13:07.154347+0000","last_clean":"2024-01-11T10:13:07.154347+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:07.154347+0000","last_undegraded":"2024-01-11T10:13:07.154347+0000","last_fullsized":"2024-01-11T10:13:07.154347+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":513,"log_dups_size":0,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33685504,"num_objects":514,"num_object_clones":0,"num_object_copies":1542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":514,"num_write_kb":32896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"31'480","reported_seq":505,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:03.045717+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:03.045717+0000","last_peered":"2024-01-11T10:13:03.045717+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:03.045717+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":480,"log_dups_size":0,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31522816,"num_objects":481,"num_object_clones":0,"num_object_copies":1443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":481,"num_write_kb":30784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":481}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"31'500","reported_seq":525,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:07.144271+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:07.144271+0000","last_peered":"2024-01-11T10:13:07.144271+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:07.144271+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":501,"num_object_clones":0,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":501,"num_write_kb":32064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":501}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"31'547","reported_seq":569,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:06.892273+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:06.892273+0000","last_peered":"2024-01-11T10:13:06.892273+0000","last_clean":"2024-01-11T10:13:06.892273+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:06.892273+0000","last_undegraded":"2024-01-11T10:13:06.892273+0000","last_fullsized":"2024-01-11T10:13:06.892273+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":548,"num_write_kb":35072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"31'522","reported_seq":544,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:06.991062+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:06.991062+0000","last_peered":"2024-01-11T10:13:06.991062+0000","last_clean":"2024-01-11T10:13:06.991062+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:06.991062+0000","last_undegraded":"2024-01-11T10:13:06.991062+0000","last_fullsized":"2024-01-11T10:13:06.991062+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34275328,"num_objects":523,"num_object_clones":0,"num_object_copies":1569,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":523,"num_write_kb":33472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"31'549","reported_seq":571,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:07.167024+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:07.167024+0000","last_peered":"2024-01-11T10:13:07.167024+0000","last_clean":"2024-01-11T10:13:07.167024+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:07.167024+0000","last_undegraded":"2024-01-11T10:13:07.167024+0000","last_fullsized":"2024-01-11T10:13:07.167024+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":549,"log_dups_size":0,"ondisk_log_size":549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36044800,"num_objects":550,"num_object_clones":0,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":550,"num_write_kb":35200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":177,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:12:48.134123+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:12:48.134123+0000","last_peered":"2024-01-11T10:12:48.134123+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:12:48.134123+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.196395+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:12:48.196395+0000","last_peered":"2024-01-11T10:12:48.196395+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:12:48.196395+0000","last_undegraded":"2024-01-11T10:12:48.196395+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"31'471","reported_seq":496,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:02.986908+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:02.986908+0000","last_peered":"2024-01-11T10:13:02.986908+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:02.986908+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30932992,"num_objects":472,"num_object_clones":0,"num_object_copies":1416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":472,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":472,"num_write_kb":30208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":472}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.158071+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:12:48.158071+0000","last_peered":"2024-01-11T10:12:48.158071+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:12:48.158071+0000","last_undegraded":"2024-01-11T10:12:48.158071+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"31'500","reported_seq":178,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:03.183187+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:03.183187+0000","last_peered":"2024-01-11T10:13:03.183187+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:03.183187+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11075584,"num_objects":169,"num_object_clones":0,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":169,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":169,"num_write_kb":10816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":169}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.142679+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:12:48.142679+0000","last_peered":"2024-01-11T10:12:48.142679+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:12:48.142679+0000","last_undegraded":"2024-01-11T10:12:48.142679+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"31'500","reported_seq":196,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:07.014426+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:07.014426+0000","last_peered":"2024-01-11T10:13:07.014426+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:07.014426+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":11968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":187}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.174832+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:12:48.174832+0000","last_peered":"2024-01-11T10:12:48.174832+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:12:48.174832+0000","last_undegraded":"2024-01-11T10:12:48.174832+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"31'518","reported_seq":543,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:06.987288+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:06.987288+0000","last_peered":"2024-01-11T10:13:06.987288+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:06.987288+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":518,"log_dups_size":0,"ondisk_log_size":518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":519,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":519,"num_write_kb":33216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":519}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.119383+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:12:48.119383+0000","last_peered":"2024-01-11T10:12:48.119383+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:12:48.119383+0000","last_undegraded":"2024-01-11T10:12:48.119383+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'470","reported_seq":495,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:03.092204+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:03.092204+0000","last_peered":"2024-01-11T10:13:03.092204+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:03.092204+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30867456,"num_objects":471,"num_object_clones":0,"num_object_copies":1413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":471,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":471,"num_write_kb":30144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":471}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":46,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.118097+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:12:48.118097+0000","last_peered":"2024-01-11T10:12:48.118097+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:12:48.118097+0000","last_undegraded":"2024-01-11T10:12:48.118097+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'462","reported_seq":484,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:03.042323+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:03.042323+0000","last_peered":"2024-01-11T10:13:03.042323+0000","last_clean":"2024-01-11T10:13:03.042323+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:03.042323+0000","last_undegraded":"2024-01-11T10:13:03.042323+0000","last_fullsized":"2024-01-11T10:13:03.042323+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161356+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:12:48.161356+0000","last_peered":"2024-01-11T10:12:48.161356+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:12:48.161356+0000","last_undegraded":"2024-01-11T10:12:48.161356+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"31'530","reported_seq":552,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:07.111802+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:07.111802+0000","last_peered":"2024-01-11T10:13:07.111802+0000","last_clean":"2024-01-11T10:13:07.111802+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:07.111802+0000","last_undegraded":"2024-01-11T10:13:07.111802+0000","last_fullsized":"2024-01-11T10:13:07.111802+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34799616,"num_objects":531,"num_object_clones":0,"num_object_copies":1593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":531,"num_write_kb":33984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161069+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:12:48.161069+0000","last_peered":"2024-01-11T10:12:48.161069+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:12:48.161069+0000","last_undegraded":"2024-01-11T10:12:48.161069+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":449773568,"num_objects":6863,"num_object_clones":0,"num_object_copies":20589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3734,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6863,"num_write_kb":439232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8119,"ondisk_log_size":8119,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98552,"kb_used_data":23152,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94273288,"statfs":{"total":96636764160,"available":96535846912,"internally_reserved":0,"allocated":23707648,"data_stored":374823457,"data_compressed":17795544,"data_compressed_allocated":23425024,"data_compressed_original":374718464,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":116472,"kb_used_data":27184,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94255368,"statfs":{"total":96636764160,"available":96517496832,"internally_reserved":0,"allocated":27836416,"data_stored":440883745,"data_compressed":20932382,"data_compressed_allocated":27553792,"data_compressed_original":440778752,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705953,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":94076,"kb_used_data":26996,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94277764,"statfs":{"total":96636764160,"available":96540430336,"internally_reserved":0,"allocated":27643904,"data_stored":438196225,"data_compressed":20802810,"data_compressed_allocated":27381760,"data_compressed_original":438108160,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:09.593 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:09.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:09.727+0000 7f72e4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:09.716 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:09.732+0000 7f72e4b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:09.870 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:09.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:10.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 82 16 8858 8842 6.73861 10.0625 0.10801 0.148294 2024-01-11T10:13:11.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 83 16 9036 9020 6.79144 11.125 0.152338 0.147135 2024-01-11T10:13:11.890 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:12.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:12.023+0000 7fafa5b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:12.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:12.029+0000 7fafa5b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:12.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 84 16 9202 9186 6.83409 10.375 0.103394 0.146207 2024-01-11T10:13:12.144 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:12.171 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:13.064 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 85 16 9386 9370 6.88898 11.5 0.0743071 0.14507 2024-01-11T10:13:13.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~50s 2024-01-11T10:13:14.065 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 86 16 9565 9549 6.93895 11.1875 0.10739 0.14405 2024-01-11T10:13:14.173 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:14.301 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:14.316+0000 7f8a7c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:14.305 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:14.322+0000 7f8a7c66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:14.451 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:14.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:14.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:13:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:14.479 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:14.607 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:14.618+0000 7f3ae96f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:14.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:14.627+0000 7f3ae96f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:15.007 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:13:15.023 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:13:15.065 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 87 16 9739 9723 6.98418 10.875 0.107722 0.143045 2024-01-11T10:13:15.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:15.151+0000 7f589c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:15.143 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:15.160+0000 7f589c79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:16.016 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:13:16.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:13:16.065 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 88 16 9897 9881 7.01702 9.875 0.0242 0.14243 2024-01-11T10:13:16.193 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:16.204+0000 7f6c276de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:16.197 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:16.214+0000 7f6c276de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:16.480 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:16.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:16.615+0000 7f9406ceb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:16.604 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:16.621+0000 7f9406ceb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:16.716 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:16.738 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:17.021 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:13:17.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:13:17.065 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 89 16 10057 10041 7.05052 10 0.0628312 0.141786 2024-01-11T10:13:17.171 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:17.181+0000 7fd6d32ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:17.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:17.191+0000 7fd6d32ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:13:18.048 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:13:18.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:18.065 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 90 15 10219 10204 7.08537 10.1875 0.113073 0.141044 2024-01-11T10:13:18.174 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.187+0000 7f242a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.193+0000 7f242a66d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.360 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:18.361 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:18.372 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-11T10:13:18.152061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":564396576,"num_objects":8605,"num_object_clones":0,"num_object_copies":25815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":8740,"num_write_kb":552486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9957,"ondisk_log_size":9957,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":401460,"kb_used_data":106772,"kb_used_omap":0,"kb_used_meta":294656,"kb_avail":377085900,"statfs":{"total":386547056640,"available":386135961600,"internally_reserved":0,"allocated":109334528,"data_stored":1731493407,"data_compressed":82206027,"data_compressed_allocated":108208128,"data_compressed_original":1731084288,"omap_allocated":0,"internal_metadata":301727744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000849"},"pg_stats":[{"pgid":"4.f","version":"33'650","reported_seq":345,"reported_epoch":34,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.049109+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:17.049109+0000","last_peered":"2024-01-11T10:13:17.049109+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:17.049109+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"33'632","reported_seq":340,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.010920+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:17.010920+0000","last_peered":"2024-01-11T10:13:17.010920+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:17.010920+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430272,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":327,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":327}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'613","reported_seq":642,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.152501+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:17.152501+0000","last_peered":"2024-01-11T10:13:17.152501+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:17.152501+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":614,"num_write_kb":39296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"33'619","reported_seq":647,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.065896+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:17.065896+0000","last_peered":"2024-01-11T10:13:17.065896+0000","last_clean":"2024-01-11T10:13:17.065896+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:17.065896+0000","last_undegraded":"2024-01-11T10:13:17.065896+0000","last_fullsized":"2024-01-11T10:13:17.065896+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":620,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":620,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"31'576","reported_seq":601,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:12.910567+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:12.910567+0000","last_peered":"2024-01-11T10:13:12.910567+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:12.910567+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":577}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"33'603","reported_seq":632,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.105165+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:17.105165+0000","last_peered":"2024-01-11T10:13:17.105165+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:17.105165+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":604,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":604,"num_write_kb":38656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"33'646","reported_seq":672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.106854+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:17.106854+0000","last_peered":"2024-01-11T10:13:17.106854+0000","last_clean":"2024-01-11T10:13:17.106854+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:17.106854+0000","last_undegraded":"2024-01-11T10:13:17.106854+0000","last_fullsized":"2024-01-11T10:13:17.106854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42401792,"num_objects":647,"num_object_clones":0,"num_object_copies":1941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":647,"num_write_kb":41408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"33'621","reported_seq":647,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:16.982684+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:16.982684+0000","last_peered":"2024-01-11T10:13:16.982684+0000","last_clean":"2024-01-11T10:13:16.982684+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:16.982684+0000","last_undegraded":"2024-01-11T10:13:16.982684+0000","last_fullsized":"2024-01-11T10:13:16.982684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40763392,"num_objects":622,"num_object_clones":0,"num_object_copies":1866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":622,"num_write_kb":39808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"33'676","reported_seq":702,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.087345+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:17.087345+0000","last_peered":"2024-01-11T10:13:17.087345+0000","last_clean":"2024-01-11T10:13:17.087345+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:17.087345+0000","last_undegraded":"2024-01-11T10:13:17.087345+0000","last_fullsized":"2024-01-11T10:13:17.087345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":677,"num_object_clones":0,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":677,"num_write_kb":43328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":183,"reported_epoch":34,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.059902+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:17.059902+0000","last_peered":"2024-01-11T10:13:17.059902+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:17.059902+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"active+undersized","last_fresh":"2024-01-11T10:13:16.311405+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:16.311405+0000","last_peered":"2024-01-11T10:13:16.311405+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:16.311405+0000","last_undegraded":"2024-01-11T10:13:16.311405+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"31'582","reported_seq":607,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:13.090931+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:13.090931+0000","last_peered":"2024-01-11T10:13:13.090931+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:13.090931+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":583,"num_write_kb":37312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":583}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.158071+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:12:48.158071+0000","last_peered":"2024-01-11T10:12:48.158071+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:12:48.158071+0000","last_undegraded":"2024-01-11T10:12:48.158071+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"31'617","reported_seq":295,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:13.106341+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:13.106341+0000","last_peered":"2024-01-11T10:13:13.106341+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:13.106341+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18743296,"num_objects":286,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":286,"num_write_kb":18304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":286}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"active+undersized","last_fresh":"2024-01-11T10:13:16.306636+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:16.306636+0000","last_peered":"2024-01-11T10:13:16.306636+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:16.306636+0000","last_undegraded":"2024-01-11T10:13:16.306636+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"33'585","reported_seq":287,"reported_epoch":34,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.060578+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:17.060578+0000","last_peered":"2024-01-11T10:13:17.060578+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:17.060578+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.174832+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:12:48.174832+0000","last_peered":"2024-01-11T10:12:48.174832+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:12:48.174832+0000","last_undegraded":"2024-01-11T10:12:48.174832+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"33'626","reported_seq":655,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:16.879140+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:16.879140+0000","last_peered":"2024-01-11T10:13:16.879140+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:16.879140+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41091072,"num_objects":627,"num_object_clones":0,"num_object_copies":1881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":627,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":627,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":627}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"active+undersized","last_fresh":"2024-01-11T10:13:17.065600+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:17.065600+0000","last_peered":"2024-01-11T10:13:17.065600+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:17.065600+0000","last_undegraded":"2024-01-11T10:13:17.065600+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'589","reported_seq":614,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:12.994675+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:12.994675+0000","last_peered":"2024-01-11T10:13:12.994675+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:12.994675+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":590,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":590,"num_write_kb":37760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":590}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":34,"state":"active+undersized","last_fresh":"2024-01-11T10:13:17.054177+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:17.054177+0000","last_peered":"2024-01-11T10:13:17.054177+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:17.054177+0000","last_undegraded":"2024-01-11T10:13:17.054177+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'598","reported_seq":620,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:13.149440+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:13.149440+0000","last_peered":"2024-01-11T10:13:13.149440+0000","last_clean":"2024-01-11T10:13:13.149440+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:13.149440+0000","last_undegraded":"2024-01-11T10:13:13.149440+0000","last_fullsized":"2024-01-11T10:13:13.149440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":38336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161356+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:12:48.161356+0000","last_peered":"2024-01-11T10:12:48.161356+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:12:48.161356+0000","last_undegraded":"2024-01-11T10:12:48.161356+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"33'627","reported_seq":653,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.073226+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:17.073226+0000","last_peered":"2024-01-11T10:13:17.073226+0000","last_clean":"2024-01-11T10:13:17.073226+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:17.073226+0000","last_undegraded":"2024-01-11T10:13:17.073226+0000","last_fullsized":"2024-01-11T10:13:17.073226+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":628,"num_write_kb":40192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"active+undersized","last_fresh":"2024-01-11T10:13:16.312061+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:16.312061+0000","last_peered":"2024-01-11T10:13:16.312061+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:16.312061+0000","last_undegraded":"2024-01-11T10:13:16.312061+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":563806208,"num_objects":8603,"num_object_clones":0,"num_object_copies":25809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":550592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9860,"ondisk_log_size":9860,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91344,"kb_used_data":28424,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94280496,"statfs":{"total":96636764160,"available":96543227904,"internally_reserved":0,"allocated":29106176,"data_stored":461199905,"data_compressed":21897160,"data_compressed_allocated":28823552,"data_compressed_original":461094912,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126412,"kb_used_data":32964,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94245428,"statfs":{"total":96636764160,"available":96507318272,"internally_reserved":0,"allocated":33755136,"data_stored":535327611,"data_compressed":25416774,"data_compressed_allocated":33456128,"data_compressed_original":535216128,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107928,"kb_used_data":32528,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94263912,"statfs":{"total":96636764160,"available":96526245888,"internally_reserved":0,"allocated":33308672,"data_stored":528576859,"data_compressed":25094258,"data_compressed_allocated":33030144,"data_compressed_original":528482304,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:18.373 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:18.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.493+0000 7f6433a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.500+0000 7f6433a64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.666 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:18.666 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:18.678 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-01-11T10:13:18.152061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":564396576,"num_objects":8605,"num_object_clones":0,"num_object_copies":25815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4812,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":8740,"num_write_kb":552486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9957,"ondisk_log_size":9957,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":401460,"kb_used_data":106772,"kb_used_omap":0,"kb_used_meta":294656,"kb_avail":377085900,"statfs":{"total":386547056640,"available":386135961600,"internally_reserved":0,"allocated":109334528,"data_stored":1731493407,"data_compressed":82206027,"data_compressed_allocated":108208128,"data_compressed_original":1731084288,"omap_allocated":0,"internal_metadata":301727744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":597,"num_write_kb":38208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000849"},"pg_stats":[{"pgid":"4.f","version":"33'650","reported_seq":345,"reported_epoch":34,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.049109+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:17.049109+0000","last_peered":"2024-01-11T10:13:17.049109+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:17.049109+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":650,"log_dups_size":0,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"33'632","reported_seq":340,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.010920+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:17.010920+0000","last_peered":"2024-01-11T10:13:17.010920+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:17.010920+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21430272,"num_objects":327,"num_object_clones":0,"num_object_copies":981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":327,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":327,"num_write_kb":20928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":327}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'613","reported_seq":642,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.152501+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:17.152501+0000","last_peered":"2024-01-11T10:13:17.152501+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:17.152501+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":614,"num_write_kb":39296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":614}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"33'619","reported_seq":647,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.065896+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:17.065896+0000","last_peered":"2024-01-11T10:13:17.065896+0000","last_clean":"2024-01-11T10:13:17.065896+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:17.065896+0000","last_undegraded":"2024-01-11T10:13:17.065896+0000","last_fullsized":"2024-01-11T10:13:17.065896+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":619,"log_dups_size":0,"ondisk_log_size":619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40632320,"num_objects":620,"num_object_clones":0,"num_object_copies":1860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":620,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"31'576","reported_seq":601,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:12.910567+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:12.910567+0000","last_peered":"2024-01-11T10:13:12.910567+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:12.910567+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37814272,"num_objects":577,"num_object_clones":0,"num_object_copies":1731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":577,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":577,"num_write_kb":36928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":577}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"33'603","reported_seq":632,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.105165+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:17.105165+0000","last_peered":"2024-01-11T10:13:17.105165+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:17.105165+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39583744,"num_objects":604,"num_object_clones":0,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":604,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":604,"num_write_kb":38656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":604}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"33'646","reported_seq":672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.106854+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:17.106854+0000","last_peered":"2024-01-11T10:13:17.106854+0000","last_clean":"2024-01-11T10:13:17.106854+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:17.106854+0000","last_undegraded":"2024-01-11T10:13:17.106854+0000","last_fullsized":"2024-01-11T10:13:17.106854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42401792,"num_objects":647,"num_object_clones":0,"num_object_copies":1941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":647,"num_write_kb":41408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"33'621","reported_seq":647,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:16.982684+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:16.982684+0000","last_peered":"2024-01-11T10:13:16.982684+0000","last_clean":"2024-01-11T10:13:16.982684+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:16.982684+0000","last_undegraded":"2024-01-11T10:13:16.982684+0000","last_fullsized":"2024-01-11T10:13:16.982684+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":621,"log_dups_size":0,"ondisk_log_size":621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40763392,"num_objects":622,"num_object_clones":0,"num_object_copies":1866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":622,"num_write_kb":39808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"33'676","reported_seq":702,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.087345+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:17.087345+0000","last_peered":"2024-01-11T10:13:17.087345+0000","last_clean":"2024-01-11T10:13:17.087345+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:17.087345+0000","last_undegraded":"2024-01-11T10:13:17.087345+0000","last_fullsized":"2024-01-11T10:13:17.087345+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44367872,"num_objects":677,"num_object_clones":0,"num_object_copies":2031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":677,"num_write_kb":43328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":183,"reported_epoch":34,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.059902+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:17.059902+0000","last_peered":"2024-01-11T10:13:17.059902+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:17.059902+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"active+undersized","last_fresh":"2024-01-11T10:13:16.311405+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:16.311405+0000","last_peered":"2024-01-11T10:13:16.311405+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:16.311405+0000","last_undegraded":"2024-01-11T10:13:16.311405+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"31'582","reported_seq":607,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:13.090931+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:13.090931+0000","last_peered":"2024-01-11T10:13:13.090931+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:13.090931+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":582,"log_dups_size":0,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207488,"num_objects":583,"num_object_clones":0,"num_object_copies":1749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":583,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":583,"num_write_kb":37312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":583}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.158071+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:12:48.158071+0000","last_peered":"2024-01-11T10:12:48.158071+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:12:48.158071+0000","last_undegraded":"2024-01-11T10:12:48.158071+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"31'617","reported_seq":295,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:13.106341+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:13.106341+0000","last_peered":"2024-01-11T10:13:13.106341+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:13.106341+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":617,"log_dups_size":0,"ondisk_log_size":617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":18743296,"num_objects":286,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":286,"num_write_kb":18304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":286}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"active+undersized","last_fresh":"2024-01-11T10:13:16.306636+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:16.306636+0000","last_peered":"2024-01-11T10:13:16.306636+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:16.306636+0000","last_undegraded":"2024-01-11T10:13:16.306636+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"33'585","reported_seq":287,"reported_epoch":34,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:17.060578+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:17.060578+0000","last_peered":"2024-01-11T10:13:17.060578+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:17.060578+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":585,"log_dups_size":0,"ondisk_log_size":585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":272}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.174832+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:12:48.174832+0000","last_peered":"2024-01-11T10:12:48.174832+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:12:48.174832+0000","last_undegraded":"2024-01-11T10:12:48.174832+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"33'626","reported_seq":655,"reported_epoch":33,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:16.879140+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:16.879140+0000","last_peered":"2024-01-11T10:13:16.879140+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:16.879140+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":626,"log_dups_size":0,"ondisk_log_size":626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41091072,"num_objects":627,"num_object_clones":0,"num_object_copies":1881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":627,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":627,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":627}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":34,"state":"active+undersized","last_fresh":"2024-01-11T10:13:17.065600+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:17.065600+0000","last_peered":"2024-01-11T10:13:17.065600+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:17.065600+0000","last_undegraded":"2024-01-11T10:13:17.065600+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"31'589","reported_seq":614,"reported_epoch":31,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:12.994675+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:12.994675+0000","last_peered":"2024-01-11T10:13:12.994675+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:12.994675+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":589,"log_dups_size":0,"ondisk_log_size":589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38666240,"num_objects":590,"num_object_clones":0,"num_object_copies":1770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":590,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":590,"num_write_kb":37760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":590}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":52,"reported_epoch":34,"state":"active+undersized","last_fresh":"2024-01-11T10:13:17.054177+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:17.054177+0000","last_peered":"2024-01-11T10:13:17.054177+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:17.054177+0000","last_undegraded":"2024-01-11T10:13:17.054177+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"31'598","reported_seq":620,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-01-11T10:13:13.149440+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:13.149440+0000","last_peered":"2024-01-11T10:13:13.149440+0000","last_clean":"2024-01-11T10:13:13.149440+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:13.149440+0000","last_undegraded":"2024-01-11T10:13:13.149440+0000","last_fullsized":"2024-01-11T10:13:13.149440+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":599,"num_write_kb":38336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":31,"state":"active+undersized","last_fresh":"2024-01-11T10:12:48.161356+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:12:48.161356+0000","last_peered":"2024-01-11T10:12:48.161356+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:12:48.161356+0000","last_undegraded":"2024-01-11T10:12:48.161356+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"33'627","reported_seq":653,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-11T10:13:17.073226+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:17.073226+0000","last_peered":"2024-01-11T10:13:17.073226+0000","last_clean":"2024-01-11T10:13:17.073226+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:17.073226+0000","last_undegraded":"2024-01-11T10:13:17.073226+0000","last_fullsized":"2024-01-11T10:13:17.073226+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":628,"num_write_kb":40192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":46,"reported_epoch":33,"state":"active+undersized","last_fresh":"2024-01-11T10:13:16.312061+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:16.312061+0000","last_peered":"2024-01-11T10:13:16.312061+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:16.312061+0000","last_undegraded":"2024-01-11T10:13:16.312061+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":563806208,"num_objects":8603,"num_object_clones":0,"num_object_copies":25809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8603,"num_write_kb":550592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9860,"ondisk_log_size":9860,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91344,"kb_used_data":28424,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94280496,"statfs":{"total":96636764160,"available":96543227904,"internally_reserved":0,"allocated":29106176,"data_stored":461199905,"data_compressed":21897160,"data_compressed_allocated":28823552,"data_compressed_original":461094912,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":126412,"kb_used_data":32964,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94245428,"statfs":{"total":96636764160,"available":96507318272,"internally_reserved":0,"allocated":33755136,"data_stored":535327611,"data_compressed":25416774,"data_compressed_allocated":33456128,"data_compressed_original":535216128,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107928,"kb_used_data":32528,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94263912,"statfs":{"total":96636764160,"available":96526245888,"internally_reserved":0,"allocated":33308672,"data_stored":528576859,"data_compressed":25094258,"data_compressed_allocated":33030144,"data_compressed_original":528482304,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:18.679 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:13:18.679 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:18.679 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:13:18.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:13:18.683 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~55s 2024-01-11T10:13:18.739 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:18.788 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.800+0000 7f64dfa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.791 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.808+0000 7f64dfa2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:18.806 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:18.808 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:18.809 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:18.809 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:13:18.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.864+0000 7fab8a308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.869+0000 7fab8a308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.900 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:18.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:18.932 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.945+0000 7fee2b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.935 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:18.952+0000 7fee2b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:18.949 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:18.949 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:18.949 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:18.950 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:18.952 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:18.953 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:18.953 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:13:19.065 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Total time run: 90.0534 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Total writes made: 10219 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Write size: 65536 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object size: 65536 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Bandwidth (MB/sec): 7.09232 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev Bandwidth: 5.12453 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max bandwidth (MB/sec): 12.375 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min bandwidth (MB/sec): 0 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Average IOPS: 113 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev IOPS: 81.9929 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max IOPS: 198 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min IOPS: 0 2024-01-11T10:13:19.067 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Average Latency(s): 0.140971 2024-01-11T10:13:19.068 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev Latency(s): 1.20484 2024-01-11T10:13:19.068 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max latency(s): 30.6859 2024-01-11T10:13:19.068 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min latency(s): 0.00944343 2024-01-11T10:13:19.082 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Cleaning up (deleting benchmark objects) 2024-01-11T10:13:19.102 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:19.114+0000 7fbf9086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:19.105 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:19.122+0000 7fbf9086d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:19.109 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:19.113 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:19.114 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:19.114 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:13:19.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:19.242+0000 7f6d03f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:19.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:19.251+0000 7f6d03f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:19.249 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:19.251 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:20.919 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:21.044 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:21.059+0000 7fb0896f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:21.048 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:21.065+0000 7fb0896f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:21.120 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:21.139 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:23.141 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:23.270 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:23.286+0000 7f9ae4f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:23.275 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:23.291+0000 7f9ae4f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:23.393 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:23.412 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:23.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:24.187 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~61s 2024-01-11T10:13:24.252 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:24.253 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:24.253 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:24.368 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:24.381+0000 7f89e1d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:24.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:24.388+0000 7f89e1d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:24.649 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:24.650 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:24.666 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-01-11T10:13:24.153173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":543949368,"num_objects":8294,"num_object_clones":0,"num_object_copies":24882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":9741,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10957,"ondisk_log_size":10957,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":392284,"kb_used_data":105916,"kb_used_omap":0,"kb_used_meta":286336,"kb_avail":377095076,"statfs":{"total":386547056640,"available":386145357824,"internally_reserved":0,"allocated":108457984,"data_stored":1716315063,"data_compressed":81485622,"data_compressed_allocated":107257856,"data_compressed_original":1715879936,"omap_allocated":0,"internal_metadata":293208064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18677784,"num_objects":286,"num_object_clones":0,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":162,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1598,"num_write_kb":60225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001457"},"pg_stats":[{"pgid":"4.f","version":"35'690","reported_seq":388,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:22.110834+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:22.110834+0000","last_peered":"2024-01-11T10:13:22.110834+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:22.110834+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20119552,"num_objects":307,"num_object_clones":0,"num_object_copies":921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":307,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":371,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":307}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'670","reported_seq":380,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:21.994185+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:21.994185+0000","last_peered":"2024-01-11T10:13:21.994185+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:21.994185+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":670,"log_dups_size":0,"ondisk_log_size":670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19857408,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":303,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":365,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":303}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'671","reported_seq":703,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:22.027136+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:22.027136+0000","last_peered":"2024-01-11T10:13:22.027136+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:22.027136+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38207512,"num_objects":584,"num_object_clones":0,"num_object_copies":1752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":584,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":672,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":584}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'673","reported_seq":703,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:21.998417+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:21.998417+0000","last_peered":"2024-01-11T10:13:21.998417+0000","last_clean":"2024-01-11T10:13:21.998417+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:21.998417+0000","last_undegraded":"2024-01-11T10:13:21.998417+0000","last_fullsized":"2024-01-11T10:13:21.998417+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38535168,"num_objects":588,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":674,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'667","reported_seq":698,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:23.175473+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:23.175473+0000","last_peered":"2024-01-11T10:13:23.175473+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:23.175473+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37486592,"num_objects":572,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":668,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":572}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'646","reported_seq":677,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:22.138785+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:22.138785+0000","last_peered":"2024-01-11T10:13:22.138785+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:22.138785+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37683200,"num_objects":575,"num_object_clones":0,"num_object_copies":1725,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":575,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":647,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":575}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'696","reported_seq":724,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:22.172607+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:22.172607+0000","last_peered":"2024-01-11T10:13:22.172607+0000","last_clean":"2024-01-11T10:13:22.172607+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:22.172607+0000","last_undegraded":"2024-01-11T10:13:22.172607+0000","last_fullsized":"2024-01-11T10:13:22.172607+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":697,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'663","reported_seq":691,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:22.080707+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:22.080707+0000","last_peered":"2024-01-11T10:13:22.080707+0000","last_clean":"2024-01-11T10:13:22.080707+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:22.080707+0000","last_undegraded":"2024-01-11T10:13:22.080707+0000","last_fullsized":"2024-01-11T10:13:22.080707+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":663,"log_dups_size":0,"ondisk_log_size":663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38928384,"num_objects":594,"num_object_clones":0,"num_object_copies":1782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":664,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'723","reported_seq":751,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:22.061304+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:22.061304+0000","last_peered":"2024-01-11T10:13:22.061304+0000","last_clean":"2024-01-11T10:13:22.061304+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:22.061304+0000","last_undegraded":"2024-01-11T10:13:22.061304+0000","last_fullsized":"2024-01-11T10:13:22.061304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42598400,"num_objects":650,"num_object_clones":0,"num_object_copies":1950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":724,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'665","reported_seq":696,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:23.095950+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:23.095950+0000","last_peered":"2024-01-11T10:13:23.095950+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:23.095950+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38535168,"num_objects":588,"num_object_clones":0,"num_object_copies":1764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":588,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":666,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":588}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'729","reported_seq":413,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:23.000885+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:23.000885+0000","last_peered":"2024-01-11T10:13:23.000885+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:23.000885+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19529728,"num_objects":298,"num_object_clones":0,"num_object_copies":894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":298}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'630","reported_seq":334,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:22.031500+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:22.031500+0000","last_peered":"2024-01-11T10:13:22.031500+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:22.031500+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16318464,"num_objects":249,"num_object_clones":0,"num_object_copies":747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":249,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":317,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":249}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'683","reported_seq":714,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:22.151231+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:22.151231+0000","last_peered":"2024-01-11T10:13:22.151231+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:22.151231+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":683,"log_dups_size":0,"ondisk_log_size":683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":684,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":596}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'671","reported_seq":702,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:22.969496+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:22.969496+0000","last_peered":"2024-01-11T10:13:22.969496+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:22.969496+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":671,"log_dups_size":0,"ondisk_log_size":671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37355520,"num_objects":570,"num_object_clones":0,"num_object_copies":1710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":570,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":672,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":570}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'704","reported_seq":732,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:23.186587+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:23.186587+0000","last_peered":"2024-01-11T10:13:23.186587+0000","last_clean":"2024-01-11T10:13:23.186587+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:23.186587+0000","last_undegraded":"2024-01-11T10:13:23.186587+0000","last_fullsized":"2024-01-11T10:13:23.186587+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39649280,"num_objects":605,"num_object_clones":0,"num_object_copies":1815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":705,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'679","reported_seq":707,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:22.184501+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:22.184501+0000","last_peered":"2024-01-11T10:13:22.184501+0000","last_clean":"2024-01-11T10:13:22.184501+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:22.184501+0000","last_undegraded":"2024-01-11T10:13:22.184501+0000","last_fullsized":"2024-01-11T10:13:22.184501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":679,"log_dups_size":0,"ondisk_log_size":679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":680,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":543359000,"num_objects":8292,"num_object_clones":0,"num_object_copies":24876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9604,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10860,"ondisk_log_size":10860,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96272,"kb_used_data":29192,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94275568,"statfs":{"total":96636764160,"available":96538181632,"internally_reserved":0,"allocated":29892608,"data_stored":473205997,"data_compressed":22467263,"data_compressed_allocated":29573120,"data_compressed_original":473088000,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":113172,"kb_used_data":32204,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94258668,"statfs":{"total":96636764160,"available":96520876032,"internally_reserved":0,"allocated":32976896,"data_stored":522620117,"data_compressed":24813526,"data_compressed_allocated":32661504,"data_compressed_original":522502144,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":107064,"kb_used_data":31664,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94264776,"statfs":{"total":96636764160,"available":96527130624,"internally_reserved":0,"allocated":32423936,"data_stored":514099917,"data_compressed":24406998,"data_compressed_allocated":32124928,"data_compressed_original":513998848,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:24.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 4.d 4.b 1.0 4.2 2024-01-11T10:13:24.812 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:24.821+0000 7f990ce64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:24.816 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:24.833+0000 7f990ce64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:25.070 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.b] on osd.0 to force-backfill; instructing pg(s) [4.2,4.d] on osd.1 to force-backfill; instructing pg(s) [1.0] on osd.3 to force-backfill; 2024-01-11T10:13:25.071 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:13:25.413 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:25.550 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:25.565+0000 7f2bfad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:25.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:25.570+0000 7f2bfad1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:25.685 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:25.709 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:27.709 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:27.825 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:27.840+0000 7f35cd75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:27.828 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:27.845+0000 7f35cd75e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:27.920 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:27.937 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:29.692 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~66s 2024-01-11T10:13:29.938 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:30.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:30.067+0000 7f8fa4d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:30.055 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:30.072+0000 7f8fa4d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:30.087 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:30.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:30.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:30.232 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:30.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:30.246+0000 7fda6706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:30.237 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:30.254+0000 7fda6706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:30.260 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:30.536 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:30.536 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:30.554 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-01-11T10:13:30.154618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":481559096,"num_objects":7342,"num_object_clones":0,"num_object_copies":22026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4045,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":10693,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11909,"ondisk_log_size":11909,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":379024,"kb_used_data":96816,"kb_used_omap":0,"kb_used_meta":282176,"kb_avail":377108336,"statfs":{"total":386547056640,"available":386158936064,"internally_reserved":0,"allocated":99139584,"data_stored":1567220663,"data_compressed":74406873,"data_compressed_allocated":97939456,"data_compressed_original":1566785536,"omap_allocated":0,"internal_metadata":288948224},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-82837480,"num_objects":-1263,"num_object_clones":0,"num_object_copies":-3789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1953,"num_write_kb":22017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002556"},"pg_stats":[{"pgid":"4.f","version":"35'744","reported_seq":442,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:27.042110+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:27.042110+0000","last_peered":"2024-01-11T10:13:27.042110+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:27.042110+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16580608,"num_objects":253,"num_object_clones":0,"num_object_copies":759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":425,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":253}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'725","reported_seq":435,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:27.087895+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:27.087895+0000","last_peered":"2024-01-11T10:13:27.087895+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:27.087895+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":725,"log_dups_size":0,"ondisk_log_size":725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16252928,"num_objects":248,"num_object_clones":0,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":420,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":248}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'731","reported_seq":763,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:26.960833+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:26.960833+0000","last_peered":"2024-01-11T10:13:26.960833+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:26.960833+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34275352,"num_objects":524,"num_object_clones":0,"num_object_copies":1572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":732,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":524}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'720","reported_seq":750,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:27.188662+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:27.188662+0000","last_peered":"2024-01-11T10:13:27.188662+0000","last_clean":"2024-01-11T10:13:27.188662+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:27.188662+0000","last_undegraded":"2024-01-11T10:13:27.188662+0000","last_fullsized":"2024-01-11T10:13:27.188662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35454976,"num_objects":541,"num_object_clones":0,"num_object_copies":1623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":721,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'720","reported_seq":751,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:28.192735+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:28.192735+0000","last_peered":"2024-01-11T10:13:28.192735+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:28.192735+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":720,"log_dups_size":0,"ondisk_log_size":720,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34013184,"num_objects":519,"num_object_clones":0,"num_object_copies":1557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":519,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":721,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":519}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'705","reported_seq":736,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:27.139978+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:27.139978+0000","last_peered":"2024-01-11T10:13:27.139978+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:27.139978+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33816576,"num_objects":516,"num_object_clones":0,"num_object_copies":1548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":516,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":706,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":516}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'759","reported_seq":787,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:27.040373+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:27.040373+0000","last_peered":"2024-01-11T10:13:27.040373+0000","last_clean":"2024-01-11T10:13:27.040373+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:27.040373+0000","last_undegraded":"2024-01-11T10:13:27.040373+0000","last_fullsized":"2024-01-11T10:13:27.040373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36306944,"num_objects":554,"num_object_clones":0,"num_object_copies":1662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":760,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'724","reported_seq":752,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:27.010109+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:27.010109+0000","last_peered":"2024-01-11T10:13:27.010109+0000","last_clean":"2024-01-11T10:13:27.010109+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:27.010109+0000","last_undegraded":"2024-01-11T10:13:27.010109+0000","last_fullsized":"2024-01-11T10:13:27.010109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34930688,"num_objects":533,"num_object_clones":0,"num_object_copies":1599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":725,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'792","reported_seq":820,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:27.086146+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:27.086146+0000","last_peered":"2024-01-11T10:13:27.086146+0000","last_clean":"2024-01-11T10:13:27.086146+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:27.086146+0000","last_undegraded":"2024-01-11T10:13:27.086146+0000","last_fullsized":"2024-01-11T10:13:27.086146+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":792,"log_dups_size":0,"ondisk_log_size":792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38076416,"num_objects":581,"num_object_clones":0,"num_object_copies":1743,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":793,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'717","reported_seq":748,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:28.060299+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:28.060299+0000","last_peered":"2024-01-11T10:13:28.060299+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:28.060299+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35127296,"num_objects":536,"num_object_clones":0,"num_object_copies":1608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":718,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":536}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'801","reported_seq":485,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:28.168917+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:28.168917+0000","last_peered":"2024-01-11T10:13:28.168917+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:28.168917+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":801,"log_dups_size":0,"ondisk_log_size":801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14811136,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":470,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'697","reported_seq":401,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:27.150962+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:27.150962+0000","last_peered":"2024-01-11T10:13:27.150962+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:27.150962+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":697,"log_dups_size":0,"ondisk_log_size":697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11927552,"num_objects":182,"num_object_clones":0,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":182,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":384,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":182}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'741","reported_seq":772,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:27.101612+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:27.101612+0000","last_peered":"2024-01-11T10:13:27.101612+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:27.101612+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35258368,"num_objects":538,"num_object_clones":0,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":742,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":538}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'740","reported_seq":771,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:27.901160+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:27.901160+0000","last_peered":"2024-01-11T10:13:27.901160+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:27.901160+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32833536,"num_objects":501,"num_object_clones":0,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":501,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":501}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'760","reported_seq":788,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:28.186941+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:28.186941+0000","last_peered":"2024-01-11T10:13:28.186941+0000","last_clean":"2024-01-11T10:13:28.186941+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:28.186941+0000","last_undegraded":"2024-01-11T10:13:28.186941+0000","last_fullsized":"2024-01-11T10:13:28.186941+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":760,"log_dups_size":0,"ondisk_log_size":760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35979264,"num_objects":549,"num_object_clones":0,"num_object_copies":1647,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":761,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'736","reported_seq":764,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:27.151635+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:27.151635+0000","last_peered":"2024-01-11T10:13:27.151635+0000","last_clean":"2024-01-11T10:13:27.151635+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:27.151635+0000","last_undegraded":"2024-01-11T10:13:27.151635+0000","last_fullsized":"2024-01-11T10:13:27.151635+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35323904,"num_objects":539,"num_object_clones":0,"num_object_copies":1617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":480968728,"num_objects":7340,"num_object_clones":0,"num_object_copies":22020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4043,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10556,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11812,"ondisk_log_size":11812,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97572,"kb_used_data":26332,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94274268,"statfs":{"total":96636764160,"available":96536850432,"internally_reserved":0,"allocated":26963968,"data_stored":426347757,"data_compressed":20242407,"data_compressed_allocated":26644480,"data_compressed_original":426229760,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":114236,"kb_used_data":29108,"kb_used_omap":0,"kb_used_meta":85120,"kb_avail":94257604,"statfs":{"total":96636764160,"available":96519786496,"internally_reserved":0,"allocated":29806592,"data_stored":471895253,"data_compressed":22405247,"data_compressed_allocated":29491200,"data_compressed_original":471777280,"omap_allocated":0,"internal_metadata":87162880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705973,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91440,"kb_used_data":28520,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94280400,"statfs":{"total":96636764160,"available":96543129600,"internally_reserved":0,"allocated":29204480,"data_stored":462588621,"data_compressed":21961384,"data_compressed_allocated":28905472,"data_compressed_original":462487552,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:30.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 4.d 4.b 4.a 2024-01-11T10:13:30.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:30.684+0000 7f159679e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:30.682 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:30.699+0000 7f159679e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:30.877 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.b] on osd.0 to force-recovery; instructing pg(s) [4.a,4.d] on osd.1 to force-recovery; 2024-01-11T10:13:30.878 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:13:32.260 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:32.373 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:32.389+0000 7fae65d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:32.376 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:32.393+0000 7fae65d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:32.531 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:32.556 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:34.557 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:34.673 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:34.688+0000 7f415426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:34.677 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:34.694+0000 7f415426d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:34.871 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:34.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:35.196 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~72s 2024-01-11T10:13:35.893 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:35.893 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:35.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:36.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:36.054+0000 7f839f6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:36.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:36.062+0000 7f839f6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:36.323 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:36.324 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:36.335 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-01-11T10:13:36.155797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":413401656,"num_objects":6302,"num_object_clones":0,"num_object_copies":18906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3357,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":11733,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12949,"ondisk_log_size":12949,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":381900,"kb_used_data":87212,"kb_used_omap":0,"kb_used_meta":294656,"kb_avail":377105460,"statfs":{"total":386547056640,"available":386155991040,"internally_reserved":0,"allocated":89305088,"data_stored":1409868727,"data_compressed":66934961,"data_compressed_allocated":88104960,"data_compressed_original":1409433600,"omap_allocated":0,"internal_metadata":301727744},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-130547712,"num_objects":-1992,"num_object_clones":0,"num_object_copies":-5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1287,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1992,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002623"},"pg_stats":[{"pgid":"4.f","version":"35'813","reported_seq":511,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.007815+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:32.007815+0000","last_peered":"2024-01-11T10:13:32.007815+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:32.007815+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":813,"log_dups_size":0,"ondisk_log_size":813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058624,"num_objects":184,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":494,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'794","reported_seq":504,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.193625+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:32.193625+0000","last_peered":"2024-01-11T10:13:32.193625+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:32.193625+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":489,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":179}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'791","reported_seq":823,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.177296+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:32.177296+0000","last_peered":"2024-01-11T10:13:32.177296+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:32.177296+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343192,"num_objects":464,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":464,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":792,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":464}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'773","reported_seq":803,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:32.116275+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:32.116275+0000","last_peered":"2024-01-11T10:13:32.116275+0000","last_clean":"2024-01-11T10:13:32.116275+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:32.116275+0000","last_undegraded":"2024-01-11T10:13:32.116275+0000","last_fullsized":"2024-01-11T10:13:32.116275+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":774,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'798","reported_seq":829,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:33.154056+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:33.154056+0000","last_peered":"2024-01-11T10:13:33.154056+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:33.154056+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":798,"log_dups_size":0,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28901376,"num_objects":441,"num_object_clones":0,"num_object_copies":1323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":799,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'791","reported_seq":822,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.205022+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:32.205022+0000","last_peered":"2024-01-11T10:13:32.205022+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:32.205022+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":791,"log_dups_size":0,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":430,"num_object_clones":0,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":430,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":792,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":430}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'821","reported_seq":849,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:31.897427+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:31.897427+0000","last_peered":"2024-01-11T10:13:31.897427+0000","last_clean":"2024-01-11T10:13:31.897427+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:31.897427+0000","last_undegraded":"2024-01-11T10:13:31.897427+0000","last_fullsized":"2024-01-11T10:13:31.897427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":821,"log_dups_size":0,"ondisk_log_size":821,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32243712,"num_objects":492,"num_object_clones":0,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":822,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'787","reported_seq":815,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:32.133874+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:32.133874+0000","last_peered":"2024-01-11T10:13:32.133874+0000","last_clean":"2024-01-11T10:13:32.133874+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:32.133874+0000","last_undegraded":"2024-01-11T10:13:32.133874+0000","last_fullsized":"2024-01-11T10:13:32.133874+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30801920,"num_objects":470,"num_object_clones":0,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'848","reported_seq":876,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:32.056221+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:32.056221+0000","last_peered":"2024-01-11T10:13:32.056221+0000","last_clean":"2024-01-11T10:13:32.056221+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:32.056221+0000","last_undegraded":"2024-01-11T10:13:32.056221+0000","last_fullsized":"2024-01-11T10:13:32.056221+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":848,"log_dups_size":0,"ondisk_log_size":848,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34406400,"num_objects":525,"num_object_clones":0,"num_object_copies":1575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":849,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'793","reported_seq":824,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:33.189843+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:33.189843+0000","last_peered":"2024-01-11T10:13:33.189843+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:33.189843+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":793,"log_dups_size":0,"ondisk_log_size":793,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30146560,"num_objects":460,"num_object_clones":0,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":460,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":794,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":460}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'866","reported_seq":550,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:33.092505+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:33.092505+0000","last_peered":"2024-01-11T10:13:33.092505+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:33.092505+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":535,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'757","reported_seq":461,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.098154+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:32.098154+0000","last_peered":"2024-01-11T10:13:32.098154+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:32.098154+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":757,"log_dups_size":0,"ondisk_log_size":757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":444,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":122}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'806","reported_seq":837,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.014668+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:32.014668+0000","last_peered":"2024-01-11T10:13:32.014668+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:32.014668+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30998528,"num_objects":473,"num_object_clones":0,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":473,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":807,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":473}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'800","reported_seq":831,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:32.971568+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:32.971568+0000","last_peered":"2024-01-11T10:13:32.971568+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:32.971568+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28901376,"num_objects":441,"num_object_clones":0,"num_object_copies":1323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":441,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":441}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'818","reported_seq":846,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:33.147260+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:33.147260+0000","last_peered":"2024-01-11T10:13:33.147260+0000","last_clean":"2024-01-11T10:13:33.147260+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:33.147260+0000","last_undegraded":"2024-01-11T10:13:33.147260+0000","last_fullsized":"2024-01-11T10:13:33.147260+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":818,"log_dups_size":0,"ondisk_log_size":818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":819,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'796","reported_seq":824,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:31.886390+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:31.886390+0000","last_peered":"2024-01-11T10:13:31.886390+0000","last_clean":"2024-01-11T10:13:31.886390+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:31.886390+0000","last_undegraded":"2024-01-11T10:13:31.886390+0000","last_fullsized":"2024-01-11T10:13:31.886390+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":797,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":412811288,"num_objects":6300,"num_object_clones":0,"num_object_copies":18900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3355,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11596,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12852,"ondisk_log_size":12852,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705979,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":98756,"kb_used_data":23356,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94273084,"statfs":{"total":96636764160,"available":96535638016,"internally_reserved":0,"allocated":23916544,"data_stored":377588973,"data_compressed":17927079,"data_compressed_allocated":23597056,"data_compressed_original":377470976,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115032,"kb_used_data":25744,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94256808,"statfs":{"total":96636764160,"available":96518971392,"internally_reserved":0,"allocated":26361856,"data_stored":416779477,"data_compressed":19788055,"data_compressed_allocated":26046464,"data_compressed_original":416661504,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705978,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92336,"kb_used_data":25256,"kb_used_omap":0,"kb_used_meta":67072,"kb_avail":94279504,"statfs":{"total":96636764160,"available":96542212096,"internally_reserved":0,"allocated":25862144,"data_stored":409111245,"data_compressed":19421992,"data_compressed_allocated":25563136,"data_compressed_original":409010176,"omap_allocated":0,"internal_metadata":68681728},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:36.336 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 4.d 4.b 4.4 4.2 4.3 2024-01-11T10:13:36.457 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:36.464+0000 7fb75ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:36.461 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:36.478+0000 7fb75ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:36.717 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.3,4.b] on osd.0 to force-backfill; instructing pg(s) [4.2,4.d] on osd.1 to force-backfill; instructing pg(s) [4.4] on osd.3 to force-backfill; 2024-01-11T10:13:36.717 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:13:36.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:37.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:37.028+0000 7fb5a7464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:37.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:37.033+0000 7fb5a7464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:37.145 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:37.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:39.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:39.288 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:39.303+0000 7f99c759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:39.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:39.308+0000 7f99c759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:39.357 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:39.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:40.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~77s 2024-01-11T10:13:41.378 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:41.495 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:41.510+0000 7f3b4b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:41.499 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:41.516+0000 7f3b4b86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:41.623 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:41.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:41.731 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:41.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:41.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:41.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:41.863+0000 7efecccde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:41.853 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:41.870+0000 7efecccde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:42.129 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:42.129 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:42.142 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-01-11T10:13:40.156578+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":349307448,"num_objects":5324,"num_object_clones":0,"num_object_copies":15972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":12711,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13927,"ondisk_log_size":13927,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":376528,"kb_used_data":77680,"kb_used_omap":0,"kb_used_meta":298816,"kb_avail":377110832,"statfs":{"total":386547056640,"available":386161491968,"internally_reserved":0,"allocated":79544320,"data_stored":1253696439,"data_compressed":59519065,"data_compressed_allocated":78344192,"data_compressed_original":1253261312,"omap_allocated":0,"internal_metadata":305987584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-152043520,"num_objects":-2320,"num_object_clones":0,"num_object_copies":-6960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2320,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002446"},"pg_stats":[{"pgid":"4.f","version":"35'882","reported_seq":580,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:36.959158+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:36.959158+0000","last_peered":"2024-01-11T10:13:36.959158+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:36.959158+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":563,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'849","reported_seq":559,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:36.960048+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:36.960048+0000","last_peered":"2024-01-11T10:13:36.960048+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:36.960048+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8126464,"num_objects":124,"num_object_clones":0,"num_object_copies":372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":124,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":544,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":124}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'847","reported_seq":879,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:37.074104+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:37.074104+0000","last_peered":"2024-01-11T10:13:37.074104+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:37.074104+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":847,"log_dups_size":0,"ondisk_log_size":847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26673176,"num_objects":408,"num_object_clones":0,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":408,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":848,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":408}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'827","reported_seq":857,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:36.822501+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:36.822501+0000","last_peered":"2024-01-11T10:13:36.822501+0000","last_clean":"2024-01-11T10:13:36.822501+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:36.822501+0000","last_undegraded":"2024-01-11T10:13:36.822501+0000","last_fullsized":"2024-01-11T10:13:36.822501+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":827,"log_dups_size":0,"ondisk_log_size":827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28442624,"num_objects":434,"num_object_clones":0,"num_object_copies":1302,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":828,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'860","reported_seq":891,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:38.222532+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:38.222532+0000","last_peered":"2024-01-11T10:13:38.222532+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:38.222532+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24838144,"num_objects":379,"num_object_clones":0,"num_object_copies":1137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":379,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":861,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":379}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'845","reported_seq":876,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:37.071286+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:37.071286+0000","last_peered":"2024-01-11T10:13:37.071286+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:37.071286+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":845,"log_dups_size":0,"ondisk_log_size":845,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":376,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":846,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":376}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'894","reported_seq":922,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:37.139764+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:37.139764+0000","last_peered":"2024-01-11T10:13:37.139764+0000","last_clean":"2024-01-11T10:13:37.139764+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:37.139764+0000","last_undegraded":"2024-01-11T10:13:37.139764+0000","last_fullsized":"2024-01-11T10:13:37.139764+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27459584,"num_objects":419,"num_object_clones":0,"num_object_copies":1257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":895,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'862","reported_seq":890,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:37.119303+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:37.119303+0000","last_peered":"2024-01-11T10:13:37.119303+0000","last_clean":"2024-01-11T10:13:37.119303+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:37.119303+0000","last_undegraded":"2024-01-11T10:13:37.119303+0000","last_fullsized":"2024-01-11T10:13:37.119303+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25886720,"num_objects":395,"num_object_clones":0,"num_object_copies":1185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'909","reported_seq":937,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:37.039604+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:37.039604+0000","last_peered":"2024-01-11T10:13:37.039604+0000","last_clean":"2024-01-11T10:13:37.039604+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:37.039604+0000","last_undegraded":"2024-01-11T10:13:37.039604+0000","last_fullsized":"2024-01-11T10:13:37.039604+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30408704,"num_objects":464,"num_object_clones":0,"num_object_copies":1392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'856","reported_seq":887,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:38.071387+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:38.071387+0000","last_peered":"2024-01-11T10:13:38.071387+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:38.071387+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":856,"log_dups_size":0,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":857,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":397}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'929","reported_seq":613,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:38.192968+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:38.192968+0000","last_peered":"2024-01-11T10:13:38.192968+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:38.192968+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":598,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":98}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'820","reported_seq":524,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:36.913807+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:36.913807+0000","last_peered":"2024-01-11T10:13:36.913807+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:36.913807+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":820,"log_dups_size":0,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":59,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":507,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":59}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'861","reported_seq":892,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:37.062190+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:37.062190+0000","last_peered":"2024-01-11T10:13:37.062190+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:37.062190+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":861,"log_dups_size":0,"ondisk_log_size":861,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27394048,"num_objects":418,"num_object_clones":0,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":418,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":862,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":418}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'852","reported_seq":883,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:38.131610+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:38.131610+0000","last_peered":"2024-01-11T10:13:38.131610+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:38.131610+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":852,"log_dups_size":0,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":389,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":853,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":389}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'879","reported_seq":907,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:37.991205+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:37.991205+0000","last_peered":"2024-01-11T10:13:37.991205+0000","last_clean":"2024-01-11T10:13:37.991205+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:37.991205+0000","last_undegraded":"2024-01-11T10:13:37.991205+0000","last_fullsized":"2024-01-11T10:13:37.991205+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28180480,"num_objects":430,"num_object_clones":0,"num_object_copies":1290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":880,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'858","reported_seq":886,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:37.165427+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:37.165427+0000","last_peered":"2024-01-11T10:13:37.165427+0000","last_clean":"2024-01-11T10:13:37.165427+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:37.165427+0000","last_undegraded":"2024-01-11T10:13:37.165427+0000","last_fullsized":"2024-01-11T10:13:37.165427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27328512,"num_objects":417,"num_object_clones":0,"num_object_copies":1251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":859,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":348717080,"num_objects":5322,"num_object_clones":0,"num_object_copies":15966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2763,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12574,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13830,"ondisk_log_size":13830,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95716,"kb_used_data":20316,"kb_used_omap":0,"kb_used_meta":75392,"kb_avail":94276124,"statfs":{"total":96636764160,"available":96538750976,"internally_reserved":0,"allocated":20803584,"data_stored":327781613,"data_compressed":15561959,"data_compressed_allocated":20484096,"data_compressed_original":327663616,"omap_allocated":0,"internal_metadata":77201408},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111772,"kb_used_data":22484,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94260068,"statfs":{"total":96636764160,"available":96522309632,"internally_reserved":0,"allocated":23023616,"data_stored":363367637,"data_compressed":17251775,"data_compressed_allocated":22708224,"data_compressed_original":363249664,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705983,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93264,"kb_used_data":22024,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94278576,"statfs":{"total":96636764160,"available":96541261824,"internally_reserved":0,"allocated":22552576,"data_stored":356158157,"data_compressed":16907496,"data_compressed_allocated":22253568,"data_compressed_original":356057088,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:42.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 4.f 4.d 4.7 4.4 2024-01-11T10:13:42.293 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:42.295+0000 7f426782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:42.297 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:42.314+0000 7f426782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:42.494 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.7] on osd.0 to force-recovery; instructing pg(s) [4.d] on osd.1 to force-recovery; instructing pg(s) [4.4,4.f] on osd.3 to force-recovery; 2024-01-11T10:13:42.494 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:13:43.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:43.786 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:43.802+0000 7fe4e779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:43.790 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:43.807+0000 7fe4e779e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:43.890 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:43.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:43.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:45.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:46.047 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:46.063+0000 7f4142213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:46.051 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:46.068+0000 7f4142213640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:46.163 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:46.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:46.205 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~83s 2024-01-11T10:13:47.509 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:47.509 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:47.509 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:13:47.631 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:47.644+0000 7fa1ebcf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:47.634 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:47.651+0000 7fa1ebcf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:47.837 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:13:47.837 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:13:47.853 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-01-11T10:13:46.157779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285999672,"num_objects":4358,"num_object_clones":0,"num_object_copies":13074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2170,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":13677,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14893,"ondisk_log_size":14893,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":358972,"kb_used_data":68444,"kb_used_omap":0,"kb_used_meta":290496,"kb_avail":377128388,"statfs":{"total":386547056640,"available":386179469312,"internally_reserved":0,"allocated":70086656,"data_stored":1102373815,"data_compressed":52333457,"data_compressed_allocated":68886528,"data_compressed_original":1101938688,"omap_allocated":0,"internal_metadata":297467904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-127401984,"num_objects":-1944,"num_object_clones":0,"num_object_copies":-5832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1944,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002376"},"pg_stats":[{"pgid":"4.f","version":"35'948","reported_seq":646,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:41.966437+0000","last_change":"2024-01-11T10:12:48.160756+0000","last_active":"2024-01-11T10:13:41.966437+0000","last_peered":"2024-01-11T10:13:41.966437+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:13:41.966437+0000","last_undegraded":"2024-01-11T10:12:48.114581+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":948,"log_dups_size":0,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3211264,"num_objects":49,"num_object_clones":0,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":49,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":629,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":49}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'905","reported_seq":615,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.065102+0000","last_change":"2024-01-11T10:12:48.262301+0000","last_active":"2024-01-11T10:13:42.065102+0000","last_peered":"2024-01-11T10:13:42.065102+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:13:42.065102+0000","last_undegraded":"2024-01-11T10:12:48.202281+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":905,"log_dups_size":0,"ondisk_log_size":905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4456448,"num_objects":68,"num_object_clones":0,"num_object_copies":204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":68,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":600,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":68}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'901","reported_seq":933,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.111588+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:13:42.111588+0000","last_peered":"2024-01-11T10:13:42.111588+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:13:42.111588+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134232,"num_objects":354,"num_object_clones":0,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":902,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":354}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'897","reported_seq":927,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:42.183169+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:13:42.183169+0000","last_peered":"2024-01-11T10:13:42.183169+0000","last_clean":"2024-01-11T10:13:42.183169+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:13:42.183169+0000","last_undegraded":"2024-01-11T10:13:42.183169+0000","last_fullsized":"2024-01-11T10:13:42.183169+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":897,"log_dups_size":0,"ondisk_log_size":897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23855104,"num_objects":364,"num_object_clones":0,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":898,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'927","reported_seq":958,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:43.100114+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:13:43.100114+0000","last_peered":"2024-01-11T10:13:43.100114+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:13:43.100114+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":928,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":312}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'909","reported_seq":940,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.020879+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:13:42.020879+0000","last_peered":"2024-01-11T10:13:42.020879+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:13:42.020879+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":909,"log_dups_size":0,"ondisk_log_size":909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20447232,"num_objects":312,"num_object_clones":0,"num_object_copies":936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":312,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":910,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":312}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'950","reported_seq":978,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:42.003890+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:13:42.003890+0000","last_peered":"2024-01-11T10:13:42.003890+0000","last_clean":"2024-01-11T10:13:42.003890+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:13:42.003890+0000","last_undegraded":"2024-01-11T10:13:42.003890+0000","last_fullsized":"2024-01-11T10:13:42.003890+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":950,"log_dups_size":0,"ondisk_log_size":950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23789568,"num_objects":363,"num_object_clones":0,"num_object_copies":1089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":951,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'917","reported_seq":945,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:42.066583+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:13:42.066583+0000","last_peered":"2024-01-11T10:13:42.066583+0000","last_clean":"2024-01-11T10:13:42.066583+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:13:42.066583+0000","last_undegraded":"2024-01-11T10:13:42.066583+0000","last_fullsized":"2024-01-11T10:13:42.066583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22282240,"num_objects":340,"num_object_clones":0,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":918,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'984","reported_seq":1012,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:42.175960+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:13:42.175960+0000","last_peered":"2024-01-11T10:13:42.175960+0000","last_clean":"2024-01-11T10:13:42.175960+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:13:42.175960+0000","last_undegraded":"2024-01-11T10:13:42.175960+0000","last_fullsized":"2024-01-11T10:13:42.175960+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":984,"log_dups_size":0,"ondisk_log_size":984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25493504,"num_objects":389,"num_object_clones":0,"num_object_copies":1167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":985,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'911","reported_seq":942,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.994476+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:13:42.994476+0000","last_peered":"2024-01-11T10:13:42.994476+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:13:42.994476+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":911,"log_dups_size":0,"ondisk_log_size":911,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22413312,"num_objects":342,"num_object_clones":0,"num_object_copies":1026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":342,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":912,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":342}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'986","reported_seq":670,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:43.199336+0000","last_change":"2024-01-11T10:12:48.426813+0000","last_active":"2024-01-11T10:13:43.199336+0000","last_peered":"2024-01-11T10:13:43.199336+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:13:43.199336+0000","last_undegraded":"2024-01-11T10:12:48.149351+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":986,"log_dups_size":0,"ondisk_log_size":986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2686976,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":655,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":41}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'875","reported_seq":579,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.176437+0000","last_change":"2024-01-11T10:12:48.363222+0000","last_active":"2024-01-11T10:13:42.176437+0000","last_peered":"2024-01-11T10:13:42.176437+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:13:42.176437+0000","last_undegraded":"2024-01-11T10:12:48.175573+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":875,"log_dups_size":0,"ondisk_log_size":875,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262144,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":562,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'919","reported_seq":950,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.155301+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:13:42.155301+0000","last_peered":"2024-01-11T10:13:42.155301+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:13:42.155301+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592960,"num_objects":360,"num_object_clones":0,"num_object_copies":1080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":360,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":360}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'915","reported_seq":946,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:42.935529+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:13:42.935529+0000","last_peered":"2024-01-11T10:13:42.935529+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:13:42.935529+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21364736,"num_objects":326,"num_object_clones":0,"num_object_copies":978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":326,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":916,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":326}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'933","reported_seq":961,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:43.138662+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:13:43.138662+0000","last_peered":"2024-01-11T10:13:43.138662+0000","last_clean":"2024-01-11T10:13:43.138662+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:13:43.138662+0000","last_undegraded":"2024-01-11T10:13:43.138662+0000","last_fullsized":"2024-01-11T10:13:43.138662+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24641536,"num_objects":376,"num_object_clones":0,"num_object_copies":1128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'919","reported_seq":947,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:13:42.115679+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:13:42.115679+0000","last_peered":"2024-01-11T10:13:42.115679+0000","last_clean":"2024-01-11T10:13:42.115679+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:13:42.115679+0000","last_undegraded":"2024-01-11T10:13:42.115679+0000","last_fullsized":"2024-01-11T10:13:42.115679+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330816,"num_objects":356,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":285409304,"num_objects":4356,"num_object_clones":0,"num_object_copies":13068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13540,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14796,"ondisk_log_size":14796,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654705989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":80284,"kb_used_data":17364,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94291556,"statfs":{"total":96636764160,"available":96554553344,"internally_reserved":0,"allocated":17780736,"data_stored":279416045,"data_compressed":13265303,"data_compressed_allocated":17461248,"data_compressed_original":279298048,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654705988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112796,"kb_used_data":19348,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94259044,"statfs":{"total":96636764160,"available":96521261056,"internally_reserved":0,"allocated":19812352,"data_stored":311987413,"data_compressed":14811967,"data_compressed_allocated":19496960,"data_compressed_original":311869440,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":90116,"kb_used_data":18876,"kb_used_omap":0,"kb_used_meta":71232,"kb_avail":94281724,"statfs":{"total":96636764160,"available":96544485376,"internally_reserved":0,"allocated":19329024,"data_stored":304581325,"data_compressed":14458352,"data_compressed_allocated":19030016,"data_compressed_original":304480256,"omap_allocated":0,"internal_metadata":72941568},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:13:48.188 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:48.304 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:48.319+0000 7fae7c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:48.308 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:48.325+0000 7fae7c62d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:48.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:48.420 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:48.420 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:50.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:50.558 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:50.574+0000 7f7a0839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:50.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:50.579+0000 7f7a0839e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:50.771 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:50.791 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:51.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~88s 2024-01-11T10:13:52.791 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:52.856 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:52.856 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:52.856 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:13:52.856 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:13:52.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:52.941+0000 7f8a3e35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:52.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:52.947+0000 7f8a3e35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:52.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:52.991+0000 7f223e060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:52.983 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.000+0000 7f223e060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.002 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:53.003 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:53.005 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:53.006 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:53.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:13:53.073 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:53.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:53.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:53.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.137+0000 7f77a4f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.146+0000 7f77a4f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.188 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:53.189 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:53.192 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:53.192 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:13:53.381 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.394+0000 7f8a38064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.386 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.404+0000 7f8a38064640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.390 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:53.394 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:53.395 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:53.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:13:53.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.549+0000 7f01b6024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.539 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:53.557+0000 7f01b6024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:53.556 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:53.557 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:53.559 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:55.095 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:55.239 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:55.255+0000 7f32ccf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:55.243 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:55.261+0000 7f32ccf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:55.380 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:55.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:57.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~94s 2024-01-11T10:13:57.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:13:57.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:57.528+0000 7f67cf35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:57.514 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:57.532+0000 7f67cf35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:57.641 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:57.660 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:13:58.561 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:13:58.561 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:13:58.563 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:13:58.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:13:58.704 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:58.718+0000 7f19df59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:58.708 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:58.726+0000 7f19df59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:58.750 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:58.751 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:58.753 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:58.753 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:58.753 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:13:58.899 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:58.912+0000 7fa63aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:58.902 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:58.920+0000 7fa63aaad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:58.987 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:58.988 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:58.990 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:58.990 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:13:59.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:59.146+0000 7f9547e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:59.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:59.154+0000 7f9547e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:59.140 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:59.144 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:59.144 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:59.144 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:13:59.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:59.296+0000 7f1a40b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:59.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:59.304+0000 7f1a40b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:59.302 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:13:59.303 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:13:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:13:59.306 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:13:59.661 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:13:59.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:59.791+0000 7fe8cbe24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:59.778 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:13:59.796+0000 7fe8cbe24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:13:59.866 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:13:59.888 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:13:59.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:01.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:02.003 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:02.020+0000 7fe868308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:02.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:02.024+0000 7fe868308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:02.142 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:02.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:02.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:02.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~99s 2024-01-11T10:14:04.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:04.285 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:04.302+0000 7f6c30d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:04.289 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:04.307+0000 7f6c30d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:04.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:04.306 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:04.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:04.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:04.455+0000 7fd7bb35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:04.447 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:04.465+0000 7fd7bb35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:04.469 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:04.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:04.678 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:04.679 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:04.695 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-01-11T10:14:04.161565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":89784888,"num_objects":1364,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":17470,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18686,"ondisk_log_size":18686,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":363088,"kb_used_data":32296,"kb_used_omap":0,"kb_used_meta":330752,"kb_avail":377124272,"statfs":{"total":386547056640,"available":386175254528,"internally_reserved":0,"allocated":33071104,"data_stored":510124983,"data_compressed":24210313,"data_compressed_allocated":31870976,"data_compressed_original":509689856,"omap_allocated":0,"internal_metadata":338690048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-139264000,"num_objects":-2125,"num_object_clones":0,"num_object_copies":-6375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2829,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002635"},"pg_stats":[{"pgid":"4.f","version":"35'1193","reported_seq":891,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:02.144601+0000","last_change":"2024-01-11T10:13:46.246978+0000","last_active":"2024-01-11T10:14:02.144601+0000","last_peered":"2024-01-11T10:14:02.144601+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:14:02.144601+0000","last_undegraded":"2024-01-11T10:14:02.144601+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1193,"log_dups_size":0,"ondisk_log_size":1193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":874,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'1155","reported_seq":865,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:02.173160+0000","last_change":"2024-01-11T10:13:47.013066+0000","last_active":"2024-01-11T10:14:02.173160+0000","last_peered":"2024-01-11T10:14:02.173160+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:14:02.173160+0000","last_undegraded":"2024-01-11T10:14:02.173160+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":850,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1127","reported_seq":1159,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:02.119584+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:14:02.119584+0000","last_peered":"2024-01-11T10:14:02.119584+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:14:02.119584+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1127,"log_dups_size":0,"ondisk_log_size":1127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323096,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1128,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":128}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'1139","reported_seq":1169,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:02.199148+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:14:02.199148+0000","last_peered":"2024-01-11T10:14:02.199148+0000","last_clean":"2024-01-11T10:14:02.199148+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:14:02.199148+0000","last_undegraded":"2024-01-11T10:14:02.199148+0000","last_fullsized":"2024-01-11T10:14:02.199148+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1139,"log_dups_size":0,"ondisk_log_size":1139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1140,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'1155","reported_seq":1186,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:02.950771+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:14:02.950771+0000","last_peered":"2024-01-11T10:14:02.950771+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:14:02.950771+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1155,"log_dups_size":0,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5505024,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1156,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":84}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'1109","reported_seq":1140,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:02.173830+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:14:02.173830+0000","last_peered":"2024-01-11T10:14:02.173830+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:14:02.173830+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1109,"log_dups_size":0,"ondisk_log_size":1109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1110,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":112}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'1203","reported_seq":1231,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:02.106455+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:14:02.106455+0000","last_peered":"2024-01-11T10:14:02.106455+0000","last_clean":"2024-01-11T10:14:02.106455+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:14:02.106455+0000","last_undegraded":"2024-01-11T10:14:02.106455+0000","last_fullsized":"2024-01-11T10:14:02.106455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1203,"log_dups_size":0,"ondisk_log_size":1203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7208960,"num_objects":110,"num_object_clones":0,"num_object_copies":330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1204,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'1150","reported_seq":1178,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:02.200523+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:14:02.200523+0000","last_peered":"2024-01-11T10:14:02.200523+0000","last_clean":"2024-01-11T10:14:02.200523+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:14:02.200523+0000","last_undegraded":"2024-01-11T10:14:02.200523+0000","last_fullsized":"2024-01-11T10:14:02.200523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1150,"log_dups_size":0,"ondisk_log_size":1150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7012352,"num_objects":107,"num_object_clones":0,"num_object_copies":321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1151,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'1233","reported_seq":1261,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:02.051362+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:14:02.051362+0000","last_peered":"2024-01-11T10:14:02.051362+0000","last_clean":"2024-01-11T10:14:02.051362+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:14:02.051362+0000","last_undegraded":"2024-01-11T10:14:02.051362+0000","last_fullsized":"2024-01-11T10:14:02.051362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1234,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'1148","reported_seq":1179,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:03.168755+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:14:03.168755+0000","last_peered":"2024-01-11T10:14:03.168755+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:14:03.168755+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6881280,"num_objects":105,"num_object_clones":0,"num_object_copies":315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":105,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1149,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":105}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'1233","reported_seq":917,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:03.135778+0000","last_change":"2024-01-11T10:13:46.269186+0000","last_active":"2024-01-11T10:14:03.135778+0000","last_peered":"2024-01-11T10:14:03.135778+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:14:03.135778+0000","last_undegraded":"2024-01-11T10:14:03.135778+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":902,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'1094","reported_seq":798,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:01.965080+0000","last_change":"2024-01-11T10:13:42.628886+0000","last_active":"2024-01-11T10:14:01.965080+0000","last_peered":"2024-01-11T10:14:01.965080+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:14:01.965080+0000","last_undegraded":"2024-01-11T10:14:01.965080+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1094,"log_dups_size":0,"ondisk_log_size":1094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":781,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'1157","reported_seq":1188,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:02.068610+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:14:02.068610+0000","last_peered":"2024-01-11T10:14:02.068610+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:14:02.068610+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1157,"log_dups_size":0,"ondisk_log_size":1157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7995392,"num_objects":122,"num_object_clones":0,"num_object_copies":366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":122,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1158,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":122}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'1147","reported_seq":1178,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:03.008661+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:14:03.008661+0000","last_peered":"2024-01-11T10:14:03.008661+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:14:03.008661+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1147,"log_dups_size":0,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1148,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":94}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'1180","reported_seq":1208,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:03.216880+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:14:03.216880+0000","last_peered":"2024-01-11T10:14:03.216880+0000","last_clean":"2024-01-11T10:14:03.216880+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:14:03.216880+0000","last_undegraded":"2024-01-11T10:14:03.216880+0000","last_fullsized":"2024-01-11T10:14:03.216880+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8454144,"num_objects":129,"num_object_clones":0,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1181,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'1166","reported_seq":1194,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:02.184109+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:14:02.184109+0000","last_peered":"2024-01-11T10:14:02.184109+0000","last_clean":"2024-01-11T10:14:02.184109+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:14:02.184109+0000","last_undegraded":"2024-01-11T10:14:02.184109+0000","last_fullsized":"2024-01-11T10:14:02.184109+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1166,"log_dups_size":0,"ondisk_log_size":1166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7143424,"num_objects":109,"num_object_clones":0,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1167,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":89194520,"num_objects":1362,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":645,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17333,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18589,"ondisk_log_size":18589,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706009,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":99264,"kb_used_data":5816,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94272576,"statfs":{"total":96636764160,"available":96535117824,"internally_reserved":0,"allocated":5955584,"data_stored":90213613,"data_compressed":4280959,"data_compressed_allocated":5636096,"data_compressed_original":90095616,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96304,"kb_used_data":7016,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94275536,"statfs":{"total":96636764160,"available":96538148864,"internally_reserved":0,"allocated":7184384,"data_stored":109939925,"data_compressed":5217671,"data_compressed_allocated":6868992,"data_compressed_original":109821952,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706008,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":91744,"kb_used_data":6608,"kb_used_omap":0,"kb_used_meta":85120,"kb_avail":94280096,"statfs":{"total":96636764160,"available":96542818304,"internally_reserved":0,"allocated":6766592,"data_stored":103582413,"data_compressed":4913848,"data_compressed_allocated":6467584,"data_compressed_original":103481344,"omap_allocated":0,"internal_metadata":87162880},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:04.696 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 1.0 4.6 4.2 4.3 2024-01-11T10:14:04.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:04.857+0000 7f668019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:04.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:04.869+0000 7f668019e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:05.057 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.3,4.6] on osd.0 to force-recovery; instructing pg(s) [4.2] on osd.1 to force-recovery; instructing pg(s) [1.0] on osd.3 to force-recovery; 2024-01-11T10:14:05.057 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:14:06.492 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:06.611 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:06.627+0000 7feb7caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:06.615 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:06.633+0000 7feb7caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:06.698 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:06.717 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:08.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~105s 2024-01-11T10:14:08.719 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:08.833 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:08.849+0000 7f014bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:08.836 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:08.854+0000 7f014bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:09.020 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:09.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:10.071 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:10.071 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:10.071 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:10.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:10.200+0000 7fc053f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:10.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:10.208+0000 7fc053f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:10.385 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:10.386 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:10.399 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-01-11T10:14:10.162848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":43057720,"num_objects":651,"num_object_clones":0,"num_object_copies":1953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":302,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":18421,"num_write_kb":574503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19637,"ondisk_log_size":19637,"up":56,"acting":56,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":366392,"kb_used_data":23128,"kb_used_omap":0,"kb_used_meta":343232,"kb_avail":377120968,"statfs":{"total":386547056640,"available":386171871232,"internally_reserved":0,"allocated":23683072,"data_stored":359916471,"data_compressed":17077609,"data_compressed_allocated":22482944,"data_compressed_original":359481344,"omap_allocated":0,"internal_metadata":351469568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-108724224,"num_objects":-1659,"num_object_clones":0,"num_object_copies":-4977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-864,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2182,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002562"},"pg_stats":[{"pgid":"4.f","version":"35'1248","reported_seq":946,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:07.011435+0000","last_change":"2024-01-11T10:13:46.246978+0000","last_active":"2024-01-11T10:14:07.011435+0000","last_peered":"2024-01-11T10:14:07.011435+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.110037+0000","last_became_peered":"2024-01-11T10:12:48.110037+0000","last_unstale":"2024-01-11T10:14:07.011435+0000","last_undegraded":"2024-01-11T10:14:07.011435+0000","last_fullsized":"2024-01-11T10:12:48.038906+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1248,"log_dups_size":0,"ondisk_log_size":1248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":21696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.e","version":"35'1216","reported_seq":926,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:07.135770+0000","last_change":"2024-01-11T10:13:47.013066+0000","last_active":"2024-01-11T10:14:07.135770+0000","last_peered":"2024-01-11T10:14:07.135770+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.163194+0000","last_became_peered":"2024-01-11T10:12:48.163194+0000","last_unstale":"2024-01-11T10:14:07.135770+0000","last_undegraded":"2024-01-11T10:14:07.135770+0000","last_fullsized":"2024-01-11T10:12:48.052117+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1216,"log_dups_size":0,"ondisk_log_size":1216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":911,"num_write_kb":21376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"35'1181","reported_seq":1213,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:06.977999+0000","last_change":"2024-01-11T10:12:48.191720+0000","last_active":"2024-01-11T10:14:06.977999+0000","last_peered":"2024-01-11T10:14:06.977999+0000","last_clean":"2024-01-11T10:12:17.483462+0000","last_became_active":"2024-01-11T10:12:48.191720+0000","last_became_peered":"2024-01-11T10:12:48.191720+0000","last_unstale":"2024-01-11T10:14:06.977999+0000","last_undegraded":"2024-01-11T10:12:48.083086+0000","last_fullsized":"2024-01-11T10:12:48.082002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1181,"log_dups_size":0,"ondisk_log_size":1181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784152,"num_objects":74,"num_object_clones":0,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":74,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1182,"num_write_kb":40129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":74}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"35'1204","reported_seq":1234,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:07.173346+0000","last_change":"2024-01-11T10:11:47.189272+0000","last_active":"2024-01-11T10:14:07.173346+0000","last_peered":"2024-01-11T10:14:07.173346+0000","last_clean":"2024-01-11T10:14:07.173346+0000","last_became_active":"2024-01-11T10:11:47.186457+0000","last_became_peered":"2024-01-11T10:11:47.186457+0000","last_unstale":"2024-01-11T10:14:07.173346+0000","last_undegraded":"2024-01-11T10:14:07.173346+0000","last_fullsized":"2024-01-11T10:14:07.173346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1204,"log_dups_size":0,"ondisk_log_size":1204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1205,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"35'1200","reported_seq":1231,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:08.118663+0000","last_change":"2024-01-11T10:12:48.198258+0000","last_active":"2024-01-11T10:14:08.118663+0000","last_peered":"2024-01-11T10:14:08.118663+0000","last_clean":"2024-01-11T10:12:17.735528+0000","last_became_active":"2024-01-11T10:12:48.198258+0000","last_became_peered":"2024-01-11T10:12:48.198258+0000","last_unstale":"2024-01-11T10:14:08.118663+0000","last_undegraded":"2024-01-11T10:12:48.062527+0000","last_fullsized":"2024-01-11T10:12:48.061860+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2555904,"num_objects":39,"num_object_clones":0,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1201,"num_write_kb":39680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":39}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"35'1168","reported_seq":1199,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:06.902127+0000","last_change":"2024-01-11T10:12:48.168427+0000","last_active":"2024-01-11T10:14:06.902127+0000","last_peered":"2024-01-11T10:14:06.902127+0000","last_clean":"2024-01-11T10:12:17.672029+0000","last_became_active":"2024-01-11T10:12:48.168427+0000","last_became_peered":"2024-01-11T10:12:48.168427+0000","last_unstale":"2024-01-11T10:14:06.902127+0000","last_undegraded":"2024-01-11T10:12:48.065137+0000","last_fullsized":"2024-01-11T10:12:48.064527+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1168,"log_dups_size":0,"ondisk_log_size":1168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3473408,"num_objects":53,"num_object_clones":0,"num_object_copies":159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1169,"num_write_kb":39104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":53}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.9","version":"35'1262","reported_seq":1290,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:07.090758+0000","last_change":"2024-01-11T10:11:47.217894+0000","last_active":"2024-01-11T10:14:07.090758+0000","last_peered":"2024-01-11T10:14:07.090758+0000","last_clean":"2024-01-11T10:14:07.090758+0000","last_became_active":"2024-01-11T10:11:47.203908+0000","last_became_peered":"2024-01-11T10:11:47.203908+0000","last_unstale":"2024-01-11T10:14:07.090758+0000","last_undegraded":"2024-01-11T10:14:07.090758+0000","last_fullsized":"2024-01-11T10:14:07.090758+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1262,"log_dups_size":0,"ondisk_log_size":1262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3342336,"num_objects":51,"num_object_clones":0,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1263,"num_write_kb":42048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"35'1200","reported_seq":1228,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:07.081019+0000","last_change":"2024-01-11T10:11:47.253710+0000","last_active":"2024-01-11T10:14:07.081019+0000","last_peered":"2024-01-11T10:14:07.081019+0000","last_clean":"2024-01-11T10:14:07.081019+0000","last_became_active":"2024-01-11T10:11:47.248118+0000","last_became_peered":"2024-01-11T10:11:47.248118+0000","last_unstale":"2024-01-11T10:14:07.081019+0000","last_undegraded":"2024-01-11T10:14:07.081019+0000","last_fullsized":"2024-01-11T10:14:07.081019+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1200,"log_dups_size":0,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3735552,"num_objects":57,"num_object_clones":0,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1201,"num_write_kb":40256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"35'1300","reported_seq":1328,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:07.127075+0000","last_change":"2024-01-11T10:11:47.186091+0000","last_active":"2024-01-11T10:14:07.127075+0000","last_peered":"2024-01-11T10:14:07.127075+0000","last_clean":"2024-01-11T10:14:07.127075+0000","last_became_active":"2024-01-11T10:11:47.181258+0000","last_became_peered":"2024-01-11T10:11:47.181258+0000","last_unstale":"2024-01-11T10:14:07.127075+0000","last_undegraded":"2024-01-11T10:14:07.127075+0000","last_fullsized":"2024-01-11T10:14:07.127075+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1300,"log_dups_size":0,"ondisk_log_size":1300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4784128,"num_objects":73,"num_object_clones":0,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1301,"num_write_kb":43968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":185,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:13:18.088788+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:13:18.088788+0000","last_peered":"2024-01-11T10:13:18.088788+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:13:18.088788+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.241832+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:13:20.241832+0000","last_peered":"2024-01-11T10:13:20.241832+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:13:20.241832+0000","last_undegraded":"2024-01-11T10:13:20.241832+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"35'1213","reported_seq":1244,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:08.069610+0000","last_change":"2024-01-11T10:12:48.155633+0000","last_active":"2024-01-11T10:14:08.069610+0000","last_peered":"2024-01-11T10:14:08.069610+0000","last_clean":"2024-01-11T10:12:17.738080+0000","last_became_active":"2024-01-11T10:12:48.155633+0000","last_became_peered":"2024-01-11T10:12:48.155633+0000","last_unstale":"2024-01-11T10:14:08.069610+0000","last_undegraded":"2024-01-11T10:12:48.072676+0000","last_fullsized":"2024-01-11T10:12:48.071970+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1213,"log_dups_size":0,"ondisk_log_size":1213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2621440,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":40,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":40128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":40}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.183774+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:13:20.183774+0000","last_peered":"2024-01-11T10:13:20.183774+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:13:20.183774+0000","last_undegraded":"2024-01-11T10:13:20.183774+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"35'1301","reported_seq":985,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:08.118663+0000","last_change":"2024-01-11T10:13:46.269186+0000","last_active":"2024-01-11T10:14:08.118663+0000","last_peered":"2024-01-11T10:14:08.118663+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.140341+0000","last_became_peered":"2024-01-11T10:12:48.140341+0000","last_unstale":"2024-01-11T10:14:08.118663+0000","last_undegraded":"2024-01-11T10:14:08.118663+0000","last_fullsized":"2024-01-11T10:12:48.065554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1301,"log_dups_size":0,"ondisk_log_size":1301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":970,"num_write_kb":22272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.256146+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:13:20.256146+0000","last_peered":"2024-01-11T10:13:20.256146+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:13:20.256146+0000","last_undegraded":"2024-01-11T10:13:20.256146+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"35'1148","reported_seq":852,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:14:06.997888+0000","last_change":"2024-01-11T10:13:42.628886+0000","last_active":"2024-01-11T10:14:06.997888+0000","last_peered":"2024-01-11T10:14:06.997888+0000","last_clean":"2024-01-11T10:11:45.990370+0000","last_became_active":"2024-01-11T10:12:48.127389+0000","last_became_peered":"2024-01-11T10:12:48.127389+0000","last_unstale":"2024-01-11T10:14:06.997888+0000","last_undegraded":"2024-01-11T10:14:06.997888+0000","last_fullsized":"2024-01-11T10:12:48.036784+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":18112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.196689+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:13:20.196689+0000","last_peered":"2024-01-11T10:13:20.196689+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:13:20.196689+0000","last_undegraded":"2024-01-11T10:13:20.196689+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"35'1214","reported_seq":1245,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:07.221515+0000","last_change":"2024-01-11T10:12:48.149488+0000","last_active":"2024-01-11T10:14:07.221515+0000","last_peered":"2024-01-11T10:14:07.221515+0000","last_clean":"2024-01-11T10:12:17.648540+0000","last_became_active":"2024-01-11T10:12:48.149488+0000","last_became_peered":"2024-01-11T10:12:48.149488+0000","last_unstale":"2024-01-11T10:14:07.221515+0000","last_undegraded":"2024-01-11T10:12:48.068023+0000","last_fullsized":"2024-01-11T10:12:48.067391+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4259840,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":65,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1215,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":65}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":17,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.059134+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:13:18.059134+0000","last_peered":"2024-01-11T10:13:18.059134+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:13:18.059134+0000","last_undegraded":"2024-01-11T10:13:18.059134+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"35'1212","reported_seq":1243,"reported_epoch":35,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:08.171694+0000","last_change":"2024-01-11T10:12:48.156943+0000","last_active":"2024-01-11T10:14:08.171694+0000","last_peered":"2024-01-11T10:14:08.171694+0000","last_clean":"2024-01-11T10:12:17.721875+0000","last_became_active":"2024-01-11T10:12:48.156943+0000","last_became_peered":"2024-01-11T10:12:48.156943+0000","last_unstale":"2024-01-11T10:14:08.171694+0000","last_undegraded":"2024-01-11T10:12:48.051545+0000","last_fullsized":"2024-01-11T10:12:48.050873+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1900544,"num_objects":29,"num_object_clones":0,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":29,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1213,"num_write_kb":39744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":29}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":54,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:18.069318+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:13:18.069318+0000","last_peered":"2024-01-11T10:13:18.069318+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:13:18.069318+0000","last_undegraded":"2024-01-11T10:13:18.069318+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"35'1257","reported_seq":1285,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:08.029222+0000","last_change":"2024-01-11T10:11:47.241894+0000","last_active":"2024-01-11T10:14:08.029222+0000","last_peered":"2024-01-11T10:14:08.029222+0000","last_clean":"2024-01-11T10:14:08.029222+0000","last_became_active":"2024-01-11T10:11:47.238040+0000","last_became_peered":"2024-01-11T10:11:47.238040+0000","last_unstale":"2024-01-11T10:14:08.029222+0000","last_undegraded":"2024-01-11T10:14:08.029222+0000","last_fullsized":"2024-01-11T10:14:08.029222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1257,"log_dups_size":0,"ondisk_log_size":1257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3407872,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1258,"num_write_kb":41920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.189594+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:13:20.189594+0000","last_peered":"2024-01-11T10:13:20.189594+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:13:20.189594+0000","last_undegraded":"2024-01-11T10:13:20.189594+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"35'1216","reported_seq":1244,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-11T10:14:07.175970+0000","last_change":"2024-01-11T10:11:47.233928+0000","last_active":"2024-01-11T10:14:07.175970+0000","last_peered":"2024-01-11T10:14:07.175970+0000","last_clean":"2024-01-11T10:14:07.175970+0000","last_became_active":"2024-01-11T10:11:47.226770+0000","last_became_peered":"2024-01-11T10:11:47.226770+0000","last_unstale":"2024-01-11T10:14:07.175970+0000","last_undegraded":"2024-01-11T10:14:07.175970+0000","last_fullsized":"2024-01-11T10:14:07.175970+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:11:45.990370+0000","last_clean_scrub_stamp":"2024-01-11T10:11:45.990370+0000","objects_scrubbed":0,"log_size":1216,"log_dups_size":0,"ondisk_log_size":1216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3866624,"num_objects":59,"num_object_clones":0,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1217,"num_write_kb":40832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":35,"state":"active+undersized","last_fresh":"2024-01-11T10:13:20.257031+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:13:20.257031+0000","last_peered":"2024-01-11T10:13:20.257031+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:13:20.257031+0000","last_undegraded":"2024-01-11T10:13:20.257031+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":16,"stat_sum":{"num_bytes":42467352,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":300,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18284,"num_write_kb":572609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19540,"ondisk_log_size":19540,"up":38,"acting":38,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706014,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100496,"kb_used_data":2888,"kb_used_omap":0,"kb_used_meta":97600,"kb_avail":94271344,"statfs":{"total":96636764160,"available":96533856256,"internally_reserved":0,"allocated":2957312,"data_stored":42241261,"data_compressed":2002975,"data_compressed_allocated":2637824,"data_compressed_original":42123264,"omap_allocated":0,"internal_metadata":99942400},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97288,"kb_used_data":3840,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94274552,"statfs":{"total":96636764160,"available":96537141248,"internally_reserved":0,"allocated":3932160,"data_stored":57904341,"data_compressed":2746743,"data_compressed_allocated":3616768,"data_compressed_original":57786368,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706013,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92832,"kb_used_data":3544,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94279008,"statfs":{"total":96636764160,"available":96541704192,"internally_reserved":0,"allocated":3629056,"data_stored":53381837,"data_compressed":2530056,"data_compressed_allocated":3330048,"data_compressed_original":53280768,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:10.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 4.b 4.a 4.2 2024-01-11T10:14:10.520 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:10.531+0000 7f5d5406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:10.523 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:10.541+0000 7f5d5406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:10.720 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [4.b] on osd.0 to force-backfill; instructing pg(s) [4.2,4.a] on osd.1 to force-backfill; 2024-01-11T10:14:10.720 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:14:11.051 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:11.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:11.216+0000 7f2d7e15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:11.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:11.221+0000 7f2d7e15e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:11.354 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:11.374 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:11.374 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:12.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Removed 10219 objects 2024-01-11T10:14:12.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Clean up completed and total clean up time :53.1501 2024-01-11T10:14:13.375 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:13.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:13.509+0000 7fb768860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:13.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:13.514+0000 7fb768860640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:13.529 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:13.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:13.726 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~110s 2024-01-11T10:14:14.437 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_1 2024-01-11T10:14:14.437 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-01-11T10:14:14.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:14.571+0000 7f6262b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:14.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:14.582+0000 7f6262b08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:14.981 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_1' removed 2024-01-11T10:14:14.994 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-01-11T10:14:14.995 INFO:tasks.radosbench:Beginning radosbench... 2024-01-11T10:14:14.995 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_2 2024-01-11T10:14:14.995 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_2 16 2024-01-11T10:14:15.106 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:15.118+0000 7fbaf8220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:15.109 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:15.127+0000 7fbaf8220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:15.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:15.690 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:15.707+0000 7f08c759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:15.693 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:15.711+0000 7f08c759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:15.725 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:15.734 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:15.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:14:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:15.746 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:15.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:15.857+0000 7fd3ff304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:15.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:15.865+0000 7fd3ff304640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:15.988 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_2' created 2024-01-11T10:14:16.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-01-11T10:14:16.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:16.142+0000 7f2ba1555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:16.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:16.157+0000 7f2ba1555640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:16.992 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-01-11T10:14:16.992 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:14:17.010 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:14:17.194 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:17.206+0000 7fc8d2ceb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:17.198 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:17.216+0000 7fc8d2ceb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:17.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:17.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:17.871+0000 7efd25f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:17.857 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:17.875+0000 7efd25f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:17.889 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:17.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:17.992 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:14:18.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:14:18.011 DEBUG:teuthology.orchestra.run.smithi040:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-01-11T10:14:18.052 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-01-11T10:14:18.052 INFO:teuthology.orchestra.run:waiting for 3000 2024-01-11T10:14:18.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:hints = 1 2024-01-11T10:14:18.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-01-11T10:14:18.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object prefix: benchmark_data_smithi040_44547 2024-01-11T10:14:18.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:14:18.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 0 0 0 0 0 0 - 0 2024-01-11T10:14:18.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:18.137+0000 7fa77855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:18.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:18.151+0000 7fa77855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:19.004 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:14:19.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:14:19.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1 16 202 186 11.6241 11.625 0.0204917 0.0824934 2024-01-11T10:14:19.186 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:19.198+0000 7f7d90864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:19.190 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:19.208+0000 7f7d90864640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:19.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~116s 2024-01-11T10:14:19.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:20.005 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:14:20.023 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:14:20.023 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:20.053 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.069+0000 7f6b968f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.058 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.076+0000 7f6b968f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2 16 380 364 11.374 11.125 0.085684 0.0860707 2024-01-11T10:14:20.158 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.172+0000 7f9bc511e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.180+0000 7f9bc511e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:20.210 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:20.377 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:20.390 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-01-11T10:14:20.165285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1049120,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":144,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":332212,"kb_used_data":13908,"kb_used_omap":0,"kb_used_meta":318272,"kb_avail":377155148,"statfs":{"total":386547056640,"available":386206871552,"internally_reserved":0,"allocated":14241792,"data_stored":207896939,"data_compressed":9857769,"data_compressed_allocated":12980224,"data_compressed_original":207437824,"omap_allocated":0,"internal_metadata":325910528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000804"},"pg_stats":[{"pgid":"5.f","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:18.228338+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:18.228338+0000","last_peered":"2024-01-11T10:14:18.228338+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:18.228338+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.169563+0000","last_change":"2024-01-11T10:14:17.162129+0000","last_active":"2024-01-11T10:14:18.169563+0000","last_peered":"2024-01-11T10:14:18.169563+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:18.169563+0000","last_undegraded":"2024-01-11T10:14:18.169563+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.165579+0000","last_change":"2024-01-11T10:14:17.181728+0000","last_active":"2024-01-11T10:14:18.165579+0000","last_peered":"2024-01-11T10:14:18.165579+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:18.165579+0000","last_undegraded":"2024-01-11T10:14:18.165579+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.101193+0000","last_change":"2024-01-11T10:14:17.100856+0000","last_active":"2024-01-11T10:14:17.101193+0000","last_peered":"2024-01-11T10:14:17.101193+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:17.101193+0000","last_undegraded":"2024-01-11T10:14:17.101193+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-11T10:14:18.163055+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:18.163055+0000","last_peered":"2024-01-11T10:14:18.163055+0000","last_clean":"2024-01-11T10:14:18.163055+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:18.163055+0000","last_undegraded":"2024-01-11T10:14:18.163055+0000","last_fullsized":"2024-01-11T10:14:18.163055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.186892+0000","last_change":"2024-01-11T10:14:17.186612+0000","last_active":"2024-01-11T10:14:17.186892+0000","last_peered":"2024-01-11T10:14:17.186892+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:17.186892+0000","last_undegraded":"2024-01-11T10:14:17.186892+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.215046+0000","last_change":"2024-01-11T10:14:17.214727+0000","last_active":"2024-01-11T10:14:17.215046+0000","last_peered":"2024-01-11T10:14:17.215046+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:17.215046+0000","last_undegraded":"2024-01-11T10:14:17.215046+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.172183+0000","last_change":"2024-01-11T10:14:17.232173+0000","last_active":"2024-01-11T10:14:18.172183+0000","last_peered":"2024-01-11T10:14:18.172183+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:18.172183+0000","last_undegraded":"2024-01-11T10:14:18.172183+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.214412+0000","last_change":"2024-01-11T10:14:17.188353+0000","last_active":"2024-01-11T10:14:18.214412+0000","last_peered":"2024-01-11T10:14:18.214412+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:18.214412+0000","last_undegraded":"2024-01-11T10:14:18.214412+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":191,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:17.055603+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:17.055603+0000","last_peered":"2024-01-11T10:14:17.055603+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:17.055603+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.052714+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:17.052714+0000","last_peered":"2024-01-11T10:14:17.052714+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:17.052714+0000","last_undegraded":"2024-01-11T10:14:17.052714+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.173027+0000","last_change":"2024-01-11T10:14:17.276409+0000","last_active":"2024-01-11T10:14:18.173027+0000","last_peered":"2024-01-11T10:14:18.173027+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:18.173027+0000","last_undegraded":"2024-01-11T10:14:18.173027+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.169004+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:18.169004+0000","last_peered":"2024-01-11T10:14:18.169004+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:18.169004+0000","last_undegraded":"2024-01-11T10:14:18.169004+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.190653+0000","last_change":"2024-01-11T10:14:17.190375+0000","last_active":"2024-01-11T10:14:17.190653+0000","last_peered":"2024-01-11T10:14:17.190653+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:17.190653+0000","last_undegraded":"2024-01-11T10:14:17.190653+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.048202+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:17.048202+0000","last_peered":"2024-01-11T10:14:17.048202+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:17.048202+0000","last_undegraded":"2024-01-11T10:14:17.048202+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.105555+0000","last_change":"2024-01-11T10:14:17.105265+0000","last_active":"2024-01-11T10:14:17.105555+0000","last_peered":"2024-01-11T10:14:17.105555+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:17.105555+0000","last_undegraded":"2024-01-11T10:14:17.105555+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":52,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.152236+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:18.152236+0000","last_peered":"2024-01-11T10:14:18.152236+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:18.152236+0000","last_undegraded":"2024-01-11T10:14:18.152236+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.126921+0000","last_change":"2024-01-11T10:14:17.126610+0000","last_active":"2024-01-11T10:14:17.126921+0000","last_peered":"2024-01-11T10:14:17.126921+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:17.126921+0000","last_undegraded":"2024-01-11T10:14:17.126921+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.037215+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:17.037215+0000","last_peered":"2024-01-11T10:14:17.037215+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:17.037215+0000","last_undegraded":"2024-01-11T10:14:17.037215+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-11T10:14:18.225960+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:18.225960+0000","last_peered":"2024-01-11T10:14:18.225960+0000","last_clean":"2024-01-11T10:14:18.225960+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:18.225960+0000","last_undegraded":"2024-01-11T10:14:18.225960+0000","last_fullsized":"2024-01-11T10:14:18.225960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.047004+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:17.047004+0000","last_peered":"2024-01-11T10:14:17.047004+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:17.047004+0000","last_undegraded":"2024-01-11T10:14:17.047004+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.174311+0000","last_change":"2024-01-11T10:14:17.173969+0000","last_active":"2024-01-11T10:14:17.174311+0000","last_peered":"2024-01-11T10:14:17.174311+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:17.174311+0000","last_undegraded":"2024-01-11T10:14:17.174311+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.174499+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:18.174499+0000","last_peered":"2024-01-11T10:14:18.174499+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:18.174499+0000","last_undegraded":"2024-01-11T10:14:18.174499+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.158521+0000","last_change":"2024-01-11T10:14:17.199177+0000","last_active":"2024-01-11T10:14:18.158521+0000","last_peered":"2024-01-11T10:14:18.158521+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:18.158521+0000","last_undegraded":"2024-01-11T10:14:18.158521+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.056015+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:17.056015+0000","last_peered":"2024-01-11T10:14:17.056015+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:17.056015+0000","last_undegraded":"2024-01-11T10:14:17.056015+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81344,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94290496,"statfs":{"total":96636764160,"available":96553467904,"internally_reserved":0,"allocated":385024,"data_stored":701831,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81336,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94290504,"statfs":{"total":96636764160,"available":96553476096,"internally_reserved":0,"allocated":376832,"data_stored":698222,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93756,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94278084,"statfs":{"total":96636764160,"available":96540758016,"internally_reserved":0,"allocated":315392,"data_stored":107854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:20.392 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:20.511 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.523+0000 7fc7dd91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.513 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.531+0000 7fc7dd91e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.718 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:20.719 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:20.732 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-01-11T10:14:20.165285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1049120,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":144,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":332212,"kb_used_data":13908,"kb_used_omap":0,"kb_used_meta":318272,"kb_avail":377155148,"statfs":{"total":386547056640,"available":386206871552,"internally_reserved":0,"allocated":14241792,"data_stored":207896939,"data_compressed":9857769,"data_compressed_allocated":12980224,"data_compressed_original":207437824,"omap_allocated":0,"internal_metadata":325910528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000804"},"pg_stats":[{"pgid":"5.f","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:18.228338+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:18.228338+0000","last_peered":"2024-01-11T10:14:18.228338+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:18.228338+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.169563+0000","last_change":"2024-01-11T10:14:17.162129+0000","last_active":"2024-01-11T10:14:18.169563+0000","last_peered":"2024-01-11T10:14:18.169563+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:18.169563+0000","last_undegraded":"2024-01-11T10:14:18.169563+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.165579+0000","last_change":"2024-01-11T10:14:17.181728+0000","last_active":"2024-01-11T10:14:18.165579+0000","last_peered":"2024-01-11T10:14:18.165579+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:18.165579+0000","last_undegraded":"2024-01-11T10:14:18.165579+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.101193+0000","last_change":"2024-01-11T10:14:17.100856+0000","last_active":"2024-01-11T10:14:17.101193+0000","last_peered":"2024-01-11T10:14:17.101193+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:17.101193+0000","last_undegraded":"2024-01-11T10:14:17.101193+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-11T10:14:18.163055+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:18.163055+0000","last_peered":"2024-01-11T10:14:18.163055+0000","last_clean":"2024-01-11T10:14:18.163055+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:18.163055+0000","last_undegraded":"2024-01-11T10:14:18.163055+0000","last_fullsized":"2024-01-11T10:14:18.163055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.186892+0000","last_change":"2024-01-11T10:14:17.186612+0000","last_active":"2024-01-11T10:14:17.186892+0000","last_peered":"2024-01-11T10:14:17.186892+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:17.186892+0000","last_undegraded":"2024-01-11T10:14:17.186892+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.215046+0000","last_change":"2024-01-11T10:14:17.214727+0000","last_active":"2024-01-11T10:14:17.215046+0000","last_peered":"2024-01-11T10:14:17.215046+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:17.215046+0000","last_undegraded":"2024-01-11T10:14:17.215046+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.172183+0000","last_change":"2024-01-11T10:14:17.232173+0000","last_active":"2024-01-11T10:14:18.172183+0000","last_peered":"2024-01-11T10:14:18.172183+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:18.172183+0000","last_undegraded":"2024-01-11T10:14:18.172183+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.214412+0000","last_change":"2024-01-11T10:14:17.188353+0000","last_active":"2024-01-11T10:14:18.214412+0000","last_peered":"2024-01-11T10:14:18.214412+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:18.214412+0000","last_undegraded":"2024-01-11T10:14:18.214412+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":191,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:17.055603+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:17.055603+0000","last_peered":"2024-01-11T10:14:17.055603+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:17.055603+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.052714+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:17.052714+0000","last_peered":"2024-01-11T10:14:17.052714+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:17.052714+0000","last_undegraded":"2024-01-11T10:14:17.052714+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.173027+0000","last_change":"2024-01-11T10:14:17.276409+0000","last_active":"2024-01-11T10:14:18.173027+0000","last_peered":"2024-01-11T10:14:18.173027+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:18.173027+0000","last_undegraded":"2024-01-11T10:14:18.173027+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.169004+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:18.169004+0000","last_peered":"2024-01-11T10:14:18.169004+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:18.169004+0000","last_undegraded":"2024-01-11T10:14:18.169004+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.190653+0000","last_change":"2024-01-11T10:14:17.190375+0000","last_active":"2024-01-11T10:14:17.190653+0000","last_peered":"2024-01-11T10:14:17.190653+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:17.190653+0000","last_undegraded":"2024-01-11T10:14:17.190653+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.048202+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:17.048202+0000","last_peered":"2024-01-11T10:14:17.048202+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:17.048202+0000","last_undegraded":"2024-01-11T10:14:17.048202+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.105555+0000","last_change":"2024-01-11T10:14:17.105265+0000","last_active":"2024-01-11T10:14:17.105555+0000","last_peered":"2024-01-11T10:14:17.105555+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:17.105555+0000","last_undegraded":"2024-01-11T10:14:17.105555+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":52,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.152236+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:18.152236+0000","last_peered":"2024-01-11T10:14:18.152236+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:18.152236+0000","last_undegraded":"2024-01-11T10:14:18.152236+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.126921+0000","last_change":"2024-01-11T10:14:17.126610+0000","last_active":"2024-01-11T10:14:17.126921+0000","last_peered":"2024-01-11T10:14:17.126921+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:17.126921+0000","last_undegraded":"2024-01-11T10:14:17.126921+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.037215+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:17.037215+0000","last_peered":"2024-01-11T10:14:17.037215+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:17.037215+0000","last_undegraded":"2024-01-11T10:14:17.037215+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-11T10:14:18.225960+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:18.225960+0000","last_peered":"2024-01-11T10:14:18.225960+0000","last_clean":"2024-01-11T10:14:18.225960+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:18.225960+0000","last_undegraded":"2024-01-11T10:14:18.225960+0000","last_fullsized":"2024-01-11T10:14:18.225960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.047004+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:17.047004+0000","last_peered":"2024-01-11T10:14:17.047004+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:17.047004+0000","last_undegraded":"2024-01-11T10:14:17.047004+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.174311+0000","last_change":"2024-01-11T10:14:17.173969+0000","last_active":"2024-01-11T10:14:17.174311+0000","last_peered":"2024-01-11T10:14:17.174311+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:17.174311+0000","last_undegraded":"2024-01-11T10:14:17.174311+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.174499+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:18.174499+0000","last_peered":"2024-01-11T10:14:18.174499+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:18.174499+0000","last_undegraded":"2024-01-11T10:14:18.174499+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.158521+0000","last_change":"2024-01-11T10:14:17.199177+0000","last_active":"2024-01-11T10:14:18.158521+0000","last_peered":"2024-01-11T10:14:18.158521+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:18.158521+0000","last_undegraded":"2024-01-11T10:14:18.158521+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.056015+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:17.056015+0000","last_peered":"2024-01-11T10:14:17.056015+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:17.056015+0000","last_undegraded":"2024-01-11T10:14:17.056015+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81344,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94290496,"statfs":{"total":96636764160,"available":96553467904,"internally_reserved":0,"allocated":385024,"data_stored":701831,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81336,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94290504,"statfs":{"total":96636764160,"available":96553476096,"internally_reserved":0,"allocated":376832,"data_stored":698222,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93756,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94278084,"statfs":{"total":96636764160,"available":96540758016,"internally_reserved":0,"allocated":315392,"data_stored":107854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:20.733 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:14:20.733 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:20.734 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:20.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.875+0000 7facaa95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:20.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:20.883+0000 7facaa95e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:21.108 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3 16 589 573 11.9365 13.0625 0.155033 0.0826727 2024-01-11T10:14:21.145 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:21.146 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:21.159 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-01-11T10:14:20.165285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1049120,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":144,"num_write_kb":2342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":332212,"kb_used_data":13908,"kb_used_omap":0,"kb_used_meta":318272,"kb_avail":377155148,"statfs":{"total":386547056640,"available":386206871552,"internally_reserved":0,"allocated":14241792,"data_stored":207896939,"data_compressed":9857769,"data_compressed_allocated":12980224,"data_compressed_original":207437824,"omap_allocated":0,"internal_metadata":325910528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000804"},"pg_stats":[{"pgid":"5.f","version":"39'2","reported_seq":12,"reported_epoch":39,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:18.228338+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:18.228338+0000","last_peered":"2024-01-11T10:14:18.228338+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:18.228338+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.169563+0000","last_change":"2024-01-11T10:14:17.162129+0000","last_active":"2024-01-11T10:14:18.169563+0000","last_peered":"2024-01-11T10:14:18.169563+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:18.169563+0000","last_undegraded":"2024-01-11T10:14:18.169563+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.165579+0000","last_change":"2024-01-11T10:14:17.181728+0000","last_active":"2024-01-11T10:14:18.165579+0000","last_peered":"2024-01-11T10:14:18.165579+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:18.165579+0000","last_undegraded":"2024-01-11T10:14:18.165579+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.101193+0000","last_change":"2024-01-11T10:14:17.100856+0000","last_active":"2024-01-11T10:14:17.101193+0000","last_peered":"2024-01-11T10:14:17.101193+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:17.101193+0000","last_undegraded":"2024-01-11T10:14:17.101193+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-11T10:14:18.163055+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:18.163055+0000","last_peered":"2024-01-11T10:14:18.163055+0000","last_clean":"2024-01-11T10:14:18.163055+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:18.163055+0000","last_undegraded":"2024-01-11T10:14:18.163055+0000","last_fullsized":"2024-01-11T10:14:18.163055+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.186892+0000","last_change":"2024-01-11T10:14:17.186612+0000","last_active":"2024-01-11T10:14:17.186892+0000","last_peered":"2024-01-11T10:14:17.186892+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:17.186892+0000","last_undegraded":"2024-01-11T10:14:17.186892+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.215046+0000","last_change":"2024-01-11T10:14:17.214727+0000","last_active":"2024-01-11T10:14:17.215046+0000","last_peered":"2024-01-11T10:14:17.215046+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:17.215046+0000","last_undegraded":"2024-01-11T10:14:17.215046+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.172183+0000","last_change":"2024-01-11T10:14:17.232173+0000","last_active":"2024-01-11T10:14:18.172183+0000","last_peered":"2024-01-11T10:14:18.172183+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:18.172183+0000","last_undegraded":"2024-01-11T10:14:18.172183+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.214412+0000","last_change":"2024-01-11T10:14:17.188353+0000","last_active":"2024-01-11T10:14:18.214412+0000","last_peered":"2024-01-11T10:14:18.214412+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:18.214412+0000","last_undegraded":"2024-01-11T10:14:18.214412+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":191,"reported_epoch":38,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:17.055603+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:17.055603+0000","last_peered":"2024-01-11T10:14:17.055603+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:17.055603+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.052714+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:17.052714+0000","last_peered":"2024-01-11T10:14:17.052714+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:17.052714+0000","last_undegraded":"2024-01-11T10:14:17.052714+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.173027+0000","last_change":"2024-01-11T10:14:17.276409+0000","last_active":"2024-01-11T10:14:18.173027+0000","last_peered":"2024-01-11T10:14:18.173027+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:18.173027+0000","last_undegraded":"2024-01-11T10:14:18.173027+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.169004+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:18.169004+0000","last_peered":"2024-01-11T10:14:18.169004+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:18.169004+0000","last_undegraded":"2024-01-11T10:14:18.169004+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.190653+0000","last_change":"2024-01-11T10:14:17.190375+0000","last_active":"2024-01-11T10:14:17.190653+0000","last_peered":"2024-01-11T10:14:17.190653+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:17.190653+0000","last_undegraded":"2024-01-11T10:14:17.190653+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.048202+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:17.048202+0000","last_peered":"2024-01-11T10:14:17.048202+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:17.048202+0000","last_undegraded":"2024-01-11T10:14:17.048202+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.105555+0000","last_change":"2024-01-11T10:14:17.105265+0000","last_active":"2024-01-11T10:14:17.105555+0000","last_peered":"2024-01-11T10:14:17.105555+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:17.105555+0000","last_undegraded":"2024-01-11T10:14:17.105555+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":52,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.152236+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:18.152236+0000","last_peered":"2024-01-11T10:14:18.152236+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:18.152236+0000","last_undegraded":"2024-01-11T10:14:18.152236+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.126921+0000","last_change":"2024-01-11T10:14:17.126610+0000","last_active":"2024-01-11T10:14:17.126921+0000","last_peered":"2024-01-11T10:14:17.126921+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:17.126921+0000","last_undegraded":"2024-01-11T10:14:17.126921+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.037215+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:17.037215+0000","last_peered":"2024-01-11T10:14:17.037215+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:17.037215+0000","last_undegraded":"2024-01-11T10:14:17.037215+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"39'1","reported_seq":13,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-01-11T10:14:18.225960+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:18.225960+0000","last_peered":"2024-01-11T10:14:18.225960+0000","last_clean":"2024-01-11T10:14:18.225960+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:18.225960+0000","last_undegraded":"2024-01-11T10:14:18.225960+0000","last_fullsized":"2024-01-11T10:14:18.225960+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.047004+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:17.047004+0000","last_peered":"2024-01-11T10:14:17.047004+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:17.047004+0000","last_undegraded":"2024-01-11T10:14:17.047004+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":7,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.174311+0000","last_change":"2024-01-11T10:14:17.173969+0000","last_active":"2024-01-11T10:14:17.174311+0000","last_peered":"2024-01-11T10:14:17.174311+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:17.174311+0000","last_undegraded":"2024-01-11T10:14:17.174311+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.174499+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:18.174499+0000","last_peered":"2024-01-11T10:14:18.174499+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:18.174499+0000","last_undegraded":"2024-01-11T10:14:18.174499+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":39,"state":"active+undersized","last_fresh":"2024-01-11T10:14:18.158521+0000","last_change":"2024-01-11T10:14:17.199177+0000","last_active":"2024-01-11T10:14:18.158521+0000","last_peered":"2024-01-11T10:14:18.158521+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:18.158521+0000","last_undegraded":"2024-01-11T10:14:18.158521+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":38,"state":"active+undersized","last_fresh":"2024-01-11T10:14:17.056015+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:17.056015+0000","last_peered":"2024-01-11T10:14:17.056015+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:17.056015+0000","last_undegraded":"2024-01-11T10:14:17.056015+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706024,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81344,"kb_used_data":376,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94290496,"statfs":{"total":96636764160,"available":96553467904,"internally_reserved":0,"allocated":385024,"data_stored":701831,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":81336,"kb_used_data":368,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94290504,"statfs":{"total":96636764160,"available":96553476096,"internally_reserved":0,"allocated":376832,"data_stored":698222,"data_compressed":29967,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706023,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":93756,"kb_used_data":308,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94278084,"statfs":{"total":96636764160,"available":96540758016,"internally_reserved":0,"allocated":315392,"data_stored":107854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:22.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 4 16 764 748 11.6865 10.9375 0.0453337 0.0846442 2024-01-11T10:14:22.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:22.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:22.339+0000 7f1edecf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:22.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:22.346+0000 7f1edecf0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:22.453 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:22.476 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:23.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 5 16 956 940 11.749 12 0.108728 0.084382 2024-01-11T10:14:24.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 6 16 1184 1168 12.1657 14.25 0.0872326 0.0814235 2024-01-11T10:14:24.477 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:24.609 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:24.625+0000 7f4f2ac2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:24.612 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:24.630+0000 7f4f2ac2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:24.711 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:24.730 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:24.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~121s 2024-01-11T10:14:25.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 7 16 1389 1373 12.2579 12.8125 0.0213554 0.0812123 2024-01-11T10:14:26.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 8 16 1602 1586 12.3896 13.3125 0.0744618 0.0803668 2024-01-11T10:14:26.160 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:26.161 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:26.161 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:14:26.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:14:26.284 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.296+0000 7f5a1a79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.305+0000 7f5a1a79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:14:26.358 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:14:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:14:26.362 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:26.362 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:14:26.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.499+0000 7fc358c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.508+0000 7fc358c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:14:26.507 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:14:26.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:14:26.511 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:26.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:14:26.638 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.650+0000 7f9bafd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.658+0000 7f9bafd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.645 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:26.648 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:26.649 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:26.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:14:26.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:26.768 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.780+0000 7fe53eb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.772 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.789+0000 7fe53eb5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:14:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:14:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:14:26.800 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:14:26.801 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:14:26.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:14:26.804 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:26.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.897+0000 7f048499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:26.902+0000 7f048499e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:26.973 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:26.992 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:26.992 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:27.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 9 16 1797 1781 12.3671 12.1875 0.0934292 0.0805627 2024-01-11T10:14:28.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 10 16 2013 1997 12.4802 13.5 0.0114188 0.0798306 2024-01-11T10:14:28.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:29.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 11 16 2225 2209 12.5501 13.25 0.0611685 0.0791276 2024-01-11T10:14:29.111 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:29.126+0000 7f6bc635e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:29.115 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:29.132+0000 7f6bc635e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:29.223 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:29.245 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:30.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 12 16 2426 2410 12.5511 12.5625 0.0577606 0.0792901 2024-01-11T10:14:30.239 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~127s 2024-01-11T10:14:31.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 13 16 2644 2628 12.6336 13.625 0.0471762 0.0789301 2024-01-11T10:14:31.246 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:31.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:31.380+0000 7f3d312de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:31.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:31.386+0000 7f3d312de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:31.489 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:31.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:31.805 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:31.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:31.806 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:14:31.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:14:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:31.939+0000 7f83ebb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:31.947+0000 7f83ebb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:14:31.980 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:14:31.983 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:14:31.984 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:31.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:14:32.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 14 16 2853 2837 12.6642 13.0625 0.0838629 0.0786909 2024-01-11T10:14:32.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:32.148+0000 7efe62c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:32.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:32.157+0000 7efe62c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:14:32.162 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:14:32.163 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:14:32.163 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:14:32.163 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:14:32.163 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:14:32.163 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:14:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:14:32.166 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:32.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:14:32.287 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:32.299+0000 7f0540a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:32.291 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:32.308+0000 7f0540a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:32.295 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:32.299 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:32.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:14:32.448 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:32.459+0000 7f5bfc115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:32.452 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:32.469+0000 7f5bfc115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:14:32.465 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:14:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:14:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:14:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:14:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:14:32.466 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:14:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:14:32.468 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:14:33.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 15 16 3067 3051 12.7115 13.375 0.10556 0.0784459 2024-01-11T10:14:33.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:33.637 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:33.653+0000 7fe66eede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:33.641 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:33.658+0000 7fe66eede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:33.736 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:33.761 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:34.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 16 16 3255 3239 12.6513 11.75 0.0950212 0.0787091 2024-01-11T10:14:35.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 17 16 3477 3461 12.7232 13.875 0.104433 0.0783946 2024-01-11T10:14:35.744 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~132s 2024-01-11T10:14:35.761 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:35.877 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:35.893+0000 7fb49d224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:35.881 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:35.898+0000 7fb49d224640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:35.958 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:35.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:36.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 18 16 3692 3676 12.7629 13.4375 0.089851 0.0781881 2024-01-11T10:14:37.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 19 16 3893 3877 12.7523 12.5625 0.109354 0.0782149 2024-01-11T10:14:37.469 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:37.469 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:37.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:37.619 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:37.631+0000 7fe53586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:37.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:37.639+0000 7fe53586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:37.833 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:37.833 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:37.850 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-01-11T10:14:36.169208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":196739616,"num_objects":2995,"num_object_clones":0,"num_object_copies":8985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2595,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":3130,"num_write_kb":193446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3074,"ondisk_log_size":3074,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":387276,"kb_used_data":38444,"kb_used_omap":0,"kb_used_meta":348800,"kb_avail":377100084,"statfs":{"total":386547056640,"available":386150486016,"internally_reserved":0,"allocated":39366656,"data_stored":608490414,"data_compressed":28875959,"data_compressed_allocated":38014976,"data_compressed_original":607993856,"omap_allocated":0,"internal_metadata":357171200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":137691136,"num_objects":2101,"num_object_clones":0,"num_object_copies":6303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1817,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2101,"num_write_kb":134464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003067"},"pg_stats":[{"pgid":"5.f","version":"41'207","reported_seq":219,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.911376+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:32.911376+0000","last_peered":"2024-01-11T10:14:32.911376+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:32.911376+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13631488,"num_objects":208,"num_object_clones":0,"num_object_copies":624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":208,"num_write_kb":13312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":208}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"41'188","reported_seq":200,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:33.205159+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:14:33.205159+0000","last_peered":"2024-01-11T10:14:33.205159+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:33.205159+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12386304,"num_objects":189,"num_object_clones":0,"num_object_copies":567,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":189,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":189,"num_write_kb":12096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":189}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"41'186","reported_seq":198,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:33.153277+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:14:33.153277+0000","last_peered":"2024-01-11T10:14:33.153277+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:33.153277+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":187,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":187,"num_write_kb":11968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":187}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"41'164","reported_seq":178,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.130159+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:14:32.130159+0000","last_peered":"2024-01-11T10:14:32.130159+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:32.130159+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":165}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"41'196","reported_seq":210,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:33.162189+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:33.162189+0000","last_peered":"2024-01-11T10:14:33.162189+0000","last_clean":"2024-01-11T10:14:33.162189+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:33.162189+0000","last_undegraded":"2024-01-11T10:14:33.162189+0000","last_fullsized":"2024-01-11T10:14:33.162189+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12910592,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":197,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"41'178","reported_seq":192,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.058295+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:14:32.058295+0000","last_peered":"2024-01-11T10:14:32.058295+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:32.058295+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":179,"num_write_kb":11456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"41'196","reported_seq":208,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.100478+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:14:32.100478+0000","last_peered":"2024-01-11T10:14:32.100478+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:32.100478+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12910592,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":197,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":197,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":197}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"41'164","reported_seq":176,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:33.024149+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:14:33.024149+0000","last_peered":"2024-01-11T10:14:33.024149+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:33.024149+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813440,"num_objects":165,"num_object_clones":0,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":165,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":165,"num_write_kb":10560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":165}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"41'192","reported_seq":204,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.987907+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:14:32.987907+0000","last_peered":"2024-01-11T10:14:32.987907+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:32.987907+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12648448,"num_objects":193,"num_object_clones":0,"num_object_copies":579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":193,"num_write_kb":12352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":193}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":197,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:20.035154+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:20.035154+0000","last_peered":"2024-01-11T10:14:20.035154+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:20.035154+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285988+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:22.285988+0000","last_peered":"2024-01-11T10:14:22.285988+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:22.285988+0000","last_undegraded":"2024-01-11T10:14:22.285988+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"41'194","reported_seq":206,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:33.167054+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:14:33.167054+0000","last_peered":"2024-01-11T10:14:33.167054+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:33.167054+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":195,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":195}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.326221+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:22.326221+0000","last_peered":"2024-01-11T10:14:22.326221+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:22.326221+0000","last_undegraded":"2024-01-11T10:14:22.326221+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"41'172","reported_seq":186,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.047139+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:14:32.047139+0000","last_peered":"2024-01-11T10:14:32.047139+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:32.047139+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":173,"num_write_kb":11072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.270692+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:22.270692+0000","last_peered":"2024-01-11T10:14:22.270692+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:22.270692+0000","last_undegraded":"2024-01-11T10:14:22.270692+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"41'163","reported_seq":177,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.114912+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:14:32.114912+0000","last_peered":"2024-01-11T10:14:32.114912+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:32.114912+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10747904,"num_objects":164,"num_object_clones":0,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":164,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":164,"num_write_kb":10496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":164}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285298+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:22.285298+0000","last_peered":"2024-01-11T10:14:22.285298+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:22.285298+0000","last_undegraded":"2024-01-11T10:14:22.285298+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"41'190","reported_seq":204,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.065907+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:14:32.065907+0000","last_peered":"2024-01-11T10:14:32.065907+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:32.065907+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12517376,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":191,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":191,"num_write_kb":12224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":191}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:20.042295+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:20.042295+0000","last_peered":"2024-01-11T10:14:20.042295+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:20.042295+0000","last_undegraded":"2024-01-11T10:14:20.042295+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"41'202","reported_seq":216,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:33.085236+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:33.085236+0000","last_peered":"2024-01-11T10:14:33.085236+0000","last_clean":"2024-01-11T10:14:33.085236+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:33.085236+0000","last_undegraded":"2024-01-11T10:14:33.085236+0000","last_fullsized":"2024-01-11T10:14:33.085236+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13303808,"num_objects":203,"num_object_clones":0,"num_object_copies":609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":203,"num_write_kb":12992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:20.091435+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:20.091435+0000","last_peered":"2024-01-11T10:14:20.091435+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:20.091435+0000","last_undegraded":"2024-01-11T10:14:20.091435+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"41'194","reported_seq":208,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:32.078620+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:14:32.078620+0000","last_peered":"2024-01-11T10:14:32.078620+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:32.078620+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12779520,"num_objects":195,"num_object_clones":0,"num_object_copies":585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":195,"num_write_kb":12480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":195}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.335164+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:22.335164+0000","last_peered":"2024-01-11T10:14:22.335164+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:22.335164+0000","last_undegraded":"2024-01-11T10:14:22.335164+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"41'191","reported_seq":203,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:33.147324+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:14:33.147324+0000","last_peered":"2024-01-11T10:14:33.147324+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:33.147324+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":192,"num_object_clones":0,"num_object_copies":576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":192,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":12288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":192}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.253192+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:22.253192+0000","last_peered":"2024-01-11T10:14:22.253192+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:22.253192+0000","last_undegraded":"2024-01-11T10:14:22.253192+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":196149248,"num_objects":2993,"num_object_clones":0,"num_object_copies":8979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2593,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2993,"num_write_kb":191552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2977,"ondisk_log_size":2977,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":103700,"kb_used_data":10252,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94268140,"statfs":{"total":96636764160,"available":96530575360,"internally_reserved":0,"allocated":10498048,"data_stored":162127074,"data_compressed":7693913,"data_compressed_allocated":10129408,"data_compressed_original":161988608,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115396,"kb_used_data":8060,"kb_used_omap":0,"kb_used_meta":107328,"kb_avail":94256444,"statfs":{"total":96636764160,"available":96518598656,"internally_reserved":0,"allocated":8253440,"data_stored":126213346,"data_compressed":5988666,"data_compressed_allocated":7884800,"data_compressed_original":126074880,"omap_allocated":0,"internal_metadata":109903872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706038,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":92404,"kb_used_data":7276,"kb_used_omap":0,"kb_used_meta":85120,"kb_avail":94279436,"statfs":{"total":96636764160,"available":96542142464,"internally_reserved":0,"allocated":7450624,"data_stored":113760962,"data_compressed":5395545,"data_compressed_allocated":7102464,"data_compressed_original":113639424,"omap_allocated":0,"internal_metadata":87162880},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:37.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:38.097 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:38.112+0000 7fdef859e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:38.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:38.118+0000 7fdef859e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:38.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:14:38.128642+0000 min lat: 0.00835493 max lat: 0.276009 avg lat: 0.0782852 2024-01-11T10:14:38.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:14:38.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 20 16 4099 4083 12.7583 12.875 0.059105 0.0782852 2024-01-11T10:14:38.215 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:38.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:39.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 21 16 4305 4289 12.7638 12.875 0.0229223 0.078146 2024-01-11T10:14:40.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 22 16 4529 4513 12.82 14 0.0172343 0.0778153 2024-01-11T10:14:40.245 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:40.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:40.378+0000 7fd55d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:40.367 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:40.384+0000 7fd55d8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:40.448 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:40.468 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:41.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 23 16 4724 4708 12.7924 12.1875 0.0516166 0.0779069 2024-01-11T10:14:41.252 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~138s 2024-01-11T10:14:42.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 24 16 4924 4908 12.7802 12.5 0.0779616 0.0780994 2024-01-11T10:14:42.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:42.606 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:42.622+0000 7f1cc339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:42.610 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:42.627+0000 7f1cc339e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:42.706 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:42.726 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:42.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:42.852 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:42.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:42.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:42.980 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:42.992+0000 7fd91739e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:42.984 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:43.002+0000 7fd91739e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:43.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 25 16 5117 5101 12.7515 12.0625 0.220957 0.0780287 2024-01-11T10:14:43.352 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:43.353 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:43.366 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":224,"stamp":"2024-01-11T10:14:42.170655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":264831520,"num_objects":4034,"num_object_clones":0,"num_object_copies":12102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3524,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":4169,"num_write_kb":259942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4113,"ondisk_log_size":4113,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":404440,"kb_used_data":47288,"kb_used_omap":0,"kb_used_meta":357120,"kb_avail":377082920,"statfs":{"total":386547056640,"available":386132910080,"internally_reserved":0,"allocated":48422912,"data_stored":753390510,"data_compressed":35756591,"data_compressed_allocated":47071232,"data_compressed_original":752893952,"omap_allocated":0,"internal_metadata":365690880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":137297920,"num_objects":2095,"num_object_clones":0,"num_object_copies":6285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1842,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2095,"num_write_kb":134080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002973"},"pg_stats":[{"pgid":"5.f","version":"41'271","reported_seq":283,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.929664+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:37.929664+0000","last_peered":"2024-01-11T10:14:37.929664+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:37.929664+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17825792,"num_objects":272,"num_object_clones":0,"num_object_copies":816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":272,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":272,"num_write_kb":17408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":272}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"41'258","reported_seq":270,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:38.199467+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:14:38.199467+0000","last_peered":"2024-01-11T10:14:38.199467+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:38.199467+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":259}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"41'258","reported_seq":270,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:38.131565+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:14:38.131565+0000","last_peered":"2024-01-11T10:14:38.131565+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:38.131565+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":259}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"41'226","reported_seq":240,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.154398+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:14:37.154398+0000","last_peered":"2024-01-11T10:14:37.154398+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:37.154398+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"41'250","reported_seq":264,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:38.088241+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:38.088241+0000","last_peered":"2024-01-11T10:14:38.088241+0000","last_clean":"2024-01-11T10:14:38.088241+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:38.088241+0000","last_undegraded":"2024-01-11T10:14:38.088241+0000","last_fullsized":"2024-01-11T10:14:38.088241+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16449536,"num_objects":251,"num_object_clones":0,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":251,"num_write_kb":16064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"41'245","reported_seq":259,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.090878+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:14:37.090878+0000","last_peered":"2024-01-11T10:14:37.090878+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:37.090878+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16121856,"num_objects":246,"num_object_clones":0,"num_object_copies":738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":246,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":246,"num_write_kb":15744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":246}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"41'258","reported_seq":270,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.111608+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:14:37.111608+0000","last_peered":"2024-01-11T10:14:37.111608+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:37.111608+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":259,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":259}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"41'226","reported_seq":238,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.986364+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:14:37.986364+0000","last_peered":"2024-01-11T10:14:37.986364+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:37.986364+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":227}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"41'261","reported_seq":273,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.954394+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:14:37.954394+0000","last_peered":"2024-01-11T10:14:37.954394+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:37.954394+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17170432,"num_objects":262,"num_object_clones":0,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":262,"num_write_kb":16768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":262}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":197,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:20.035154+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:20.035154+0000","last_peered":"2024-01-11T10:14:20.035154+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:20.035154+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285988+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:22.285988+0000","last_peered":"2024-01-11T10:14:22.285988+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:22.285988+0000","last_undegraded":"2024-01-11T10:14:22.285988+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"41'262","reported_seq":274,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:38.136071+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:14:38.136071+0000","last_peered":"2024-01-11T10:14:38.136071+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:38.136071+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17235968,"num_objects":263,"num_object_clones":0,"num_object_copies":789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":263,"num_write_kb":16832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":263}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.326221+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:22.326221+0000","last_peered":"2024-01-11T10:14:22.326221+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:22.326221+0000","last_undegraded":"2024-01-11T10:14:22.326221+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"41'230","reported_seq":244,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.138096+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:14:37.138096+0000","last_peered":"2024-01-11T10:14:37.138096+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:37.138096+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":231,"num_write_kb":14784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":231}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.270692+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:22.270692+0000","last_peered":"2024-01-11T10:14:22.270692+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:22.270692+0000","last_undegraded":"2024-01-11T10:14:22.270692+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"41'226","reported_seq":240,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.150570+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:14:37.150570+0000","last_peered":"2024-01-11T10:14:37.150570+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:37.150570+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14876672,"num_objects":227,"num_object_clones":0,"num_object_copies":681,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":227,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":227,"num_write_kb":14528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":227}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285298+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:22.285298+0000","last_peered":"2024-01-11T10:14:22.285298+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:22.285298+0000","last_undegraded":"2024-01-11T10:14:22.285298+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"41'268","reported_seq":282,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.147872+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:14:37.147872+0000","last_peered":"2024-01-11T10:14:37.147872+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:37.147872+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17629184,"num_objects":269,"num_object_clones":0,"num_object_copies":807,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":269,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":269,"num_write_kb":17216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":269}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:20.042295+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:20.042295+0000","last_peered":"2024-01-11T10:14:20.042295+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:20.042295+0000","last_undegraded":"2024-01-11T10:14:20.042295+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"41'258","reported_seq":272,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:37.950146+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:37.950146+0000","last_peered":"2024-01-11T10:14:37.950146+0000","last_clean":"2024-01-11T10:14:37.950146+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:37.950146+0000","last_undegraded":"2024-01-11T10:14:37.950146+0000","last_fullsized":"2024-01-11T10:14:37.950146+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16973824,"num_objects":259,"num_object_clones":0,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":259,"num_write_kb":16576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:20.091435+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:20.091435+0000","last_peered":"2024-01-11T10:14:20.091435+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:20.091435+0000","last_undegraded":"2024-01-11T10:14:20.091435+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"41'265","reported_seq":279,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.166289+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:14:37.166289+0000","last_peered":"2024-01-11T10:14:37.166289+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:37.166289+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17432576,"num_objects":266,"num_object_clones":0,"num_object_copies":798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":266,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":266,"num_write_kb":17024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":266}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.335164+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:22.335164+0000","last_peered":"2024-01-11T10:14:22.335164+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:22.335164+0000","last_undegraded":"2024-01-11T10:14:22.335164+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"41'254","reported_seq":266,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:37.988110+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:14:37.988110+0000","last_peered":"2024-01-11T10:14:37.988110+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:37.988110+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16711680,"num_objects":255,"num_object_clones":0,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":255,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":255,"num_write_kb":16320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":255}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.253192+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:22.253192+0000","last_peered":"2024-01-11T10:14:22.253192+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:22.253192+0000","last_undegraded":"2024-01-11T10:14:22.253192+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":264241152,"num_objects":4032,"num_object_clones":0,"num_object_copies":12096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3522,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4032,"num_write_kb":258048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4016,"ondisk_log_size":4016,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":111200,"kb_used_data":13592,"kb_used_omap":0,"kb_used_meta":97600,"kb_avail":94260640,"statfs":{"total":96636764160,"available":96522895360,"internally_reserved":0,"allocated":13918208,"data_stored":216849634,"data_compressed":10292433,"data_compressed_allocated":13549568,"data_compressed_original":216711168,"omap_allocated":0,"internal_metadata":99942400},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":122448,"kb_used_data":10952,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94249392,"statfs":{"total":96636764160,"available":96511377408,"internally_reserved":0,"allocated":11214848,"data_stored":173595874,"data_compressed":8238642,"data_compressed_allocated":10846208,"data_compressed_original":173457408,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706043,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":95016,"kb_used_data":9888,"kb_used_omap":0,"kb_used_meta":85120,"kb_avail":94276824,"statfs":{"total":96636764160,"available":96539467776,"internally_reserved":0,"allocated":10125312,"data_stored":156555970,"data_compressed":7427681,"data_compressed_allocated":9777152,"data_compressed_original":156434432,"omap_allocated":0,"internal_metadata":87162880},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:43.367 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 5.e 5.d 5.9 5.8 1.0 5.7 5.1 5.0 2024-01-11T10:14:43.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:43.501+0000 7f3e83c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:43.500 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:43.517+0000 7f3e83c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:43.732 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.0,5.7,5.8,5.d,5.e] on osd.0 to force-recovery; instructing pg(s) [5.9] on osd.1 to force-recovery; instructing pg(s) [1.0,5.1] on osd.3 to force-recovery; 2024-01-11T10:14:43.732 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:14:44.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 26 16 5294 5278 12.6865 11.0625 0.0528974 0.0786901 2024-01-11T10:14:44.727 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:44.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:44.860+0000 7fa05c11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:44.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:44.865+0000 7fa05c11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:44.961 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:44.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:45.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 27 16 5496 5480 12.6841 12.625 0.0834664 0.0786819 2024-01-11T10:14:46.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 28 16 5704 5688 12.6954 13 0.0270943 0.0786114 2024-01-11T10:14:46.757 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~143s 2024-01-11T10:14:46.980 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:47.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 29 16 5911 5895 12.7037 12.9375 0.0176287 0.0786119 2024-01-11T10:14:47.119 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:47.135+0000 7f3938508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:47.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:47.141+0000 7f3938508640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:47.250 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:47.278 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:47.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:48.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 30 16 6108 6092 12.6906 12.3125 0.0952109 0.0786898 2024-01-11T10:14:48.746 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:48.746 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:48.747 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:48.875 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:48.887+0000 7f6db14de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:48.878 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:48.896+0000 7f6db14de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:49.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 31 16 6305 6289 12.6784 12.3125 0.0621305 0.0787122 2024-01-11T10:14:49.120 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:49.121 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:49.134 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-01-11T10:14:48.172057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":359006752,"num_objects":5471,"num_object_clones":0,"num_object_copies":16413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":5606,"num_write_kb":351910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5550,"ondisk_log_size":5550,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":397324,"kb_used_data":60972,"kb_used_omap":0,"kb_used_meta":336320,"kb_avail":377090036,"statfs":{"total":386547056640,"available":386140196864,"internally_reserved":0,"allocated":62435328,"data_stored":977589166,"data_compressed":46402743,"data_compressed_allocated":61083648,"data_compressed_original":977092608,"omap_allocated":0,"internal_metadata":344391680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":162267136,"num_objects":2476,"num_object_clones":0,"num_object_copies":7428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2256,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2476,"num_write_kb":158464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002848"},"pg_stats":[{"pgid":"5.f","version":"41'344","reported_seq":356,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:42.988193+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:42.988193+0000","last_peered":"2024-01-11T10:14:42.988193+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:42.988193+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22609920,"num_objects":345,"num_object_clones":0,"num_object_copies":1035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":345,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":345,"num_write_kb":22080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":345}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"41'329","reported_seq":341,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:43.076455+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:14:43.076455+0000","last_peered":"2024-01-11T10:14:43.076455+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:43.076455+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"41'329","reported_seq":341,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:43.200387+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:14:43.200387+0000","last_peered":"2024-01-11T10:14:43.200387+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:43.200387+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21626880,"num_objects":330,"num_object_clones":0,"num_object_copies":990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":330,"num_write_kb":21120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":330}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"41'348","reported_seq":362,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:47.110630+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:14:47.110630+0000","last_peered":"2024-01-11T10:14:47.110630+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:47.110630+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22872064,"num_objects":349,"num_object_clones":0,"num_object_copies":1047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":349,"num_write_kb":22336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":349}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"41'305","reported_seq":319,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:43.145444+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:43.145444+0000","last_peered":"2024-01-11T10:14:43.145444+0000","last_clean":"2024-01-11T10:14:43.145444+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:43.145444+0000","last_undegraded":"2024-01-11T10:14:43.145444+0000","last_fullsized":"2024-01-11T10:14:43.145444+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20054016,"num_objects":306,"num_object_clones":0,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":19584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"41'351","reported_seq":365,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:46.998341+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:14:46.998341+0000","last_peered":"2024-01-11T10:14:46.998341+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:46.998341+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23068672,"num_objects":352,"num_object_clones":0,"num_object_copies":1056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":352,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":352,"num_write_kb":22528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":352}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"41'396","reported_seq":408,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:47.102334+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:14:47.102334+0000","last_peered":"2024-01-11T10:14:47.102334+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:47.102334+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26017792,"num_objects":397,"num_object_clones":0,"num_object_copies":1191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":397,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":397,"num_write_kb":25408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":397}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"41'267","reported_seq":279,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:43.113847+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:14:43.113847+0000","last_peered":"2024-01-11T10:14:43.113847+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:43.113847+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17563648,"num_objects":268,"num_object_clones":0,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":268,"num_write_kb":17152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":268}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"41'337","reported_seq":349,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:43.087666+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:14:43.087666+0000","last_peered":"2024-01-11T10:14:43.087666+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:43.087666+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22151168,"num_objects":338,"num_object_clones":0,"num_object_copies":1014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":338,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":338,"num_write_kb":21632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":338}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":197,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:20.035154+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:20.035154+0000","last_peered":"2024-01-11T10:14:20.035154+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:20.035154+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285988+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:22.285988+0000","last_peered":"2024-01-11T10:14:22.285988+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:22.285988+0000","last_undegraded":"2024-01-11T10:14:22.285988+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"41'323","reported_seq":335,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:43.088922+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:14:43.088922+0000","last_peered":"2024-01-11T10:14:43.088922+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:43.088922+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21233664,"num_objects":324,"num_object_clones":0,"num_object_copies":972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":324,"num_write_kb":20736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":324}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.326221+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:22.326221+0000","last_peered":"2024-01-11T10:14:22.326221+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:22.326221+0000","last_undegraded":"2024-01-11T10:14:22.326221+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"41'357","reported_seq":371,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:47.093277+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:14:47.093277+0000","last_peered":"2024-01-11T10:14:47.093277+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:47.093277+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23461888,"num_objects":358,"num_object_clones":0,"num_object_copies":1074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":358,"num_write_kb":22912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":358}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.270692+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:22.270692+0000","last_peered":"2024-01-11T10:14:22.270692+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:22.270692+0000","last_undegraded":"2024-01-11T10:14:22.270692+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"41'346","reported_seq":360,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:47.013942+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:14:47.013942+0000","last_peered":"2024-01-11T10:14:47.013942+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:47.013942+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":347,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":347,"num_write_kb":22208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":347}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285298+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:22.285298+0000","last_peered":"2024-01-11T10:14:22.285298+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:22.285298+0000","last_undegraded":"2024-01-11T10:14:22.285298+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"41'400","reported_seq":414,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:47.101177+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:14:47.101177+0000","last_peered":"2024-01-11T10:14:47.101177+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:47.101177+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":401,"num_object_clones":0,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":401,"num_write_kb":25664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":401}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":29,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:20.042295+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:20.042295+0000","last_peered":"2024-01-11T10:14:20.042295+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:20.042295+0000","last_undegraded":"2024-01-11T10:14:20.042295+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"41'313","reported_seq":327,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:42.765211+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:42.765211+0000","last_peered":"2024-01-11T10:14:42.765211+0000","last_clean":"2024-01-11T10:14:42.765211+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:42.765211+0000","last_undegraded":"2024-01-11T10:14:42.765211+0000","last_fullsized":"2024-01-11T10:14:42.765211+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20578304,"num_objects":314,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":314,"num_write_kb":20096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":66,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:20.091435+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:20.091435+0000","last_peered":"2024-01-11T10:14:20.091435+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:20.091435+0000","last_undegraded":"2024-01-11T10:14:20.091435+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"41'387","reported_seq":401,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:47.128297+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:14:47.128297+0000","last_peered":"2024-01-11T10:14:47.128297+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:47.128297+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25427968,"num_objects":388,"num_object_clones":0,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":388,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":388,"num_write_kb":24832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":388}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.335164+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:22.335164+0000","last_peered":"2024-01-11T10:14:22.335164+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:22.335164+0000","last_undegraded":"2024-01-11T10:14:22.335164+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"41'321","reported_seq":333,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:42.956027+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:14:42.956027+0000","last_peered":"2024-01-11T10:14:42.956027+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:42.956027+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":321,"log_dups_size":0,"ondisk_log_size":321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21102592,"num_objects":322,"num_object_clones":0,"num_object_copies":966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":322,"num_write_kb":20608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":322}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.253192+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:22.253192+0000","last_peered":"2024-01-11T10:14:22.253192+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:22.253192+0000","last_undegraded":"2024-01-11T10:14:22.253192+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":358416384,"num_objects":5469,"num_object_clones":0,"num_object_copies":16407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4849,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5469,"num_write_kb":350016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5453,"ondisk_log_size":5453,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706049,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":97916,"kb_used_data":16948,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94273924,"statfs":{"total":96636764160,"available":96536498176,"internally_reserved":0,"allocated":17354752,"data_stored":271834338,"data_compressed":12903401,"data_compressed_allocated":16986112,"data_compressed_original":271695872,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706053,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":115368,"kb_used_data":16352,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94256472,"statfs":{"total":96636764160,"available":96518627328,"internally_reserved":0,"allocated":16744448,"data_stored":262069474,"data_compressed":12439842,"data_compressed_allocated":16375808,"data_compressed_original":261931008,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706053,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":108264,"kb_used_data":14816,"kb_used_omap":0,"kb_used_meta":93440,"kb_avail":94263576,"statfs":{"total":96636764160,"available":96525901824,"internally_reserved":0,"allocated":15171584,"data_stored":237296322,"data_compressed":11261665,"data_compressed_allocated":14823424,"data_compressed_original":237174784,"omap_allocated":0,"internal_metadata":95682560},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:49.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:49.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:49.413+0000 7f7b25e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:49.400 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:49.418+0000 7f7b25e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:49.519 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:49.543 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:50.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 32 16 6488 6472 12.6396 11.4375 0.111164 0.0789877 2024-01-11T10:14:51.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 33 16 6688 6672 12.6353 12.5 0.0462107 0.0789917 2024-01-11T10:14:51.544 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:51.659 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:51.675+0000 7f392be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:51.662 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:51.680+0000 7f392be6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:51.771 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:51.789 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:51.789 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:52.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 34 16 6895 6879 12.6442 12.9375 0.00905242 0.0789862 2024-01-11T10:14:52.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~149s 2024-01-11T10:14:53.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 35 16 7099 7083 12.6472 12.75 0.058094 0.0789016 2024-01-11T10:14:53.790 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:53.910 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:53.926+0000 7f48e046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:53.914 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:53.931+0000 7f48e046d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:54.025 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:54.049 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:54.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 36 16 7304 7288 12.6517 12.8125 0.0837952 0.0789548 2024-01-11T10:14:54.135 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:14:54.136 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:14:54.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:54.265+0000 7f231cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:54.260 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:54.277+0000 7f231cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:55.012 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:14:55.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:14:55.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 37 16 7492 7476 12.6273 11.75 0.137857 0.0790767 2024-01-11T10:14:55.150 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:55.161+0000 7f08fe660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:55.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:55.170+0000 7f08fe660640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:56.013 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:14:56.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:14:56.050 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:14:56.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 38 16 7684 7668 12.6108 12 0.0775208 0.0791537 2024-01-11T10:14:56.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:56.188+0000 7ff238b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:56.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:56.199+0000 7ff238b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:56.200 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:56.216+0000 7f51ed79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:56.204 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:56.221+0000 7f51ed79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:56.444 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:56.464 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:56.465 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:57.022 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:14:57.036 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:14:57.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 39 16 7855 7839 12.5615 10.6875 0.10096 0.0795178 2024-01-11T10:14:57.177 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:57.187+0000 7f3d99f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:57.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:57.198+0000 7f3d99f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:57.767 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~154s 2024-01-11T10:14:58.026 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:14:58.045 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:14:58.046 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:58.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:14:58.130335+0000 min lat: 0.00835493 max lat: 0.30334 avg lat: 0.0797656 2024-01-11T10:14:58.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:14:58.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 40 16 8031 8015 12.5224 11 0.0854144 0.0797656 2024-01-11T10:14:58.164 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.177+0000 7f0f8ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.186+0000 7f0f8ce6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.372 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:58.373 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:58.385 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-01-11T10:14:58.174663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":487719456,"num_objects":7435,"num_object_clones":0,"num_object_copies":22305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7570,"num_write_kb":477606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7514,"ondisk_log_size":7514,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":413896,"kb_used_data":77544,"kb_used_omap":0,"kb_used_meta":336320,"kb_avail":377073464,"statfs":{"total":386547056640,"available":386123227136,"internally_reserved":0,"allocated":79405056,"data_stored":1248593506,"data_compressed":59270863,"data_compressed_allocated":78020608,"data_compressed_original":1248083968,"omap_allocated":0,"internal_metadata":344391680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27000832,"num_objects":412,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001228"},"pg_stats":[{"pgid":"5.f","version":"41'494","reported_seq":506,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.174688+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:53.174688+0000","last_peered":"2024-01-11T10:14:53.174688+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:53.174688+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":495,"num_write_kb":31680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":495}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"41'462","reported_seq":474,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.019587+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:14:53.019587+0000","last_peered":"2024-01-11T10:14:53.019587+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:53.019587+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":463}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"41'431","reported_seq":443,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.151475+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:14:53.151475+0000","last_peered":"2024-01-11T10:14:53.151475+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:53.151475+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":432}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"43'465","reported_seq":485,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.066148+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:14:57.066148+0000","last_peered":"2024-01-11T10:14:57.066148+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:57.066148+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"41'403","reported_seq":417,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:53.202119+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:53.202119+0000","last_peered":"2024-01-11T10:14:53.202119+0000","last_clean":"2024-01-11T10:14:53.202119+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:53.202119+0000","last_undegraded":"2024-01-11T10:14:53.202119+0000","last_fullsized":"2024-01-11T10:14:53.202119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"44'487","reported_seq":507,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.132428+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:14:57.132428+0000","last_peered":"2024-01-11T10:14:57.132428+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:57.132428+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"44'517","reported_seq":533,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.175181+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:14:57.175181+0000","last_peered":"2024-01-11T10:14:57.175181+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:57.175181+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":518,"num_write_kb":33152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":518}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"41'397","reported_seq":409,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:52.918962+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:14:52.918962+0000","last_peered":"2024-01-11T10:14:52.918962+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:52.918962+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"41'485","reported_seq":497,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.197986+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:14:53.197986+0000","last_peered":"2024-01-11T10:14:53.197986+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:53.197986+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":203,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.077237+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:57.077237+0000","last_peered":"2024-01-11T10:14:57.077237+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:57.077237+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.127029+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:57.127029+0000","last_peered":"2024-01-11T10:14:57.127029+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:57.127029+0000","last_undegraded":"2024-01-11T10:14:57.127029+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"41'441","reported_seq":453,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:52.730006+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:14:52.730006+0000","last_peered":"2024-01-11T10:14:52.730006+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:52.730006+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":28288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":442}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.326221+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:22.326221+0000","last_peered":"2024-01-11T10:14:22.326221+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:22.326221+0000","last_undegraded":"2024-01-11T10:14:22.326221+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"44'478","reported_seq":498,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.083210+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:14:57.083210+0000","last_peered":"2024-01-11T10:14:57.083210+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:57.083210+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":479,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.171098+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:57.171098+0000","last_peered":"2024-01-11T10:14:57.171098+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:57.171098+0000","last_undegraded":"2024-01-11T10:14:57.171098+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"43'460","reported_seq":480,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.062532+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:14:57.062532+0000","last_peered":"2024-01-11T10:14:57.062532+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:57.062532+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":461,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285298+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:22.285298+0000","last_peered":"2024-01-11T10:14:22.285298+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:22.285298+0000","last_undegraded":"2024-01-11T10:14:22.285298+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"44'531","reported_seq":551,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.167121+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:14:57.167121+0000","last_peered":"2024-01-11T10:14:57.167121+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:57.167121+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.067209+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:57.067209+0000","last_peered":"2024-01-11T10:14:57.067209+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:57.067209+0000","last_undegraded":"2024-01-11T10:14:57.067209+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"41'415","reported_seq":429,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:52.955365+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:52.955365+0000","last_peered":"2024-01-11T10:14:52.955365+0000","last_clean":"2024-01-11T10:14:52.955365+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:52.955365+0000","last_undegraded":"2024-01-11T10:14:52.955365+0000","last_fullsized":"2024-01-11T10:14:52.955365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.061633+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:57.061633+0000","last_peered":"2024-01-11T10:14:57.061633+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:57.061633+0000","last_undegraded":"2024-01-11T10:14:57.061633+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"44'490","reported_seq":510,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.089192+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:14:57.089192+0000","last_peered":"2024-01-11T10:14:57.089192+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:57.089192+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":31424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.335164+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:22.335164+0000","last_peered":"2024-01-11T10:14:22.335164+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:22.335164+0000","last_undegraded":"2024-01-11T10:14:22.335164+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"41'461","reported_seq":473,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.248650+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:14:53.248650+0000","last_peered":"2024-01-11T10:14:53.248650+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:53.248650+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277632,"num_objects":462,"num_object_clones":0,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":462,"num_write_kb":29568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.177876+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:57.177876+0000","last_peered":"2024-01-11T10:14:57.177876+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:57.177876+0000","last_undegraded":"2024-01-11T10:14:57.177876+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":487129088,"num_objects":7433,"num_object_clones":0,"num_object_copies":22299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7433,"num_write_kb":475712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7417,"ondisk_log_size":7417,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112652,"kb_used_data":23364,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94259188,"statfs":{"total":96636764160,"available":96521408512,"internally_reserved":0,"allocated":23924736,"data_stored":376954082,"data_compressed":17895049,"data_compressed_allocated":23556096,"data_compressed_original":376815616,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124752,"kb_used_data":21576,"kb_used_omap":0,"kb_used_meta":103168,"kb_avail":94247088,"statfs":{"total":96636764160,"available":96509018112,"internally_reserved":0,"allocated":22093824,"data_stored":347403836,"data_compressed":16491666,"data_compressed_allocated":21708800,"data_compressed_original":347258880,"omap_allocated":0,"internal_metadata":105644032},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100716,"kb_used_data":19748,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94271124,"statfs":{"total":96636764160,"available":96533630976,"internally_reserved":0,"allocated":20221952,"data_stored":317846556,"data_compressed":15086313,"data_compressed_allocated":19857408,"data_compressed_original":317718528,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:58.387 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:58.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:14:58.504 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.517+0000 7feca0613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.509 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.526+0000 7feca0613640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.613+0000 7f7db799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.618+0000 7f7db799e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.714 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:14:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:14:58.734 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:14:58.757 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:58.757 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:58.770 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-01-11T10:14:58.174663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":487719456,"num_objects":7435,"num_object_clones":0,"num_object_copies":22305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7570,"num_write_kb":477606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7514,"ondisk_log_size":7514,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":413896,"kb_used_data":77544,"kb_used_omap":0,"kb_used_meta":336320,"kb_avail":377073464,"statfs":{"total":386547056640,"available":386123227136,"internally_reserved":0,"allocated":79405056,"data_stored":1248593506,"data_compressed":59270863,"data_compressed_allocated":78020608,"data_compressed_original":1248083968,"omap_allocated":0,"internal_metadata":344391680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27000832,"num_objects":412,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001228"},"pg_stats":[{"pgid":"5.f","version":"41'494","reported_seq":506,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.174688+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:53.174688+0000","last_peered":"2024-01-11T10:14:53.174688+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:53.174688+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":495,"num_write_kb":31680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":495}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"41'462","reported_seq":474,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.019587+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:14:53.019587+0000","last_peered":"2024-01-11T10:14:53.019587+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:53.019587+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":463}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"41'431","reported_seq":443,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.151475+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:14:53.151475+0000","last_peered":"2024-01-11T10:14:53.151475+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:53.151475+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":432}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"43'465","reported_seq":485,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.066148+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:14:57.066148+0000","last_peered":"2024-01-11T10:14:57.066148+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:57.066148+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"41'403","reported_seq":417,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:53.202119+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:53.202119+0000","last_peered":"2024-01-11T10:14:53.202119+0000","last_clean":"2024-01-11T10:14:53.202119+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:53.202119+0000","last_undegraded":"2024-01-11T10:14:53.202119+0000","last_fullsized":"2024-01-11T10:14:53.202119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"44'487","reported_seq":507,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.132428+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:14:57.132428+0000","last_peered":"2024-01-11T10:14:57.132428+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:57.132428+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"44'517","reported_seq":533,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.175181+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:14:57.175181+0000","last_peered":"2024-01-11T10:14:57.175181+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:57.175181+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":518,"num_write_kb":33152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":518}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"41'397","reported_seq":409,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:52.918962+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:14:52.918962+0000","last_peered":"2024-01-11T10:14:52.918962+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:52.918962+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"41'485","reported_seq":497,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.197986+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:14:53.197986+0000","last_peered":"2024-01-11T10:14:53.197986+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:53.197986+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":203,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.077237+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:57.077237+0000","last_peered":"2024-01-11T10:14:57.077237+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:57.077237+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.127029+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:57.127029+0000","last_peered":"2024-01-11T10:14:57.127029+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:57.127029+0000","last_undegraded":"2024-01-11T10:14:57.127029+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"41'441","reported_seq":453,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:52.730006+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:14:52.730006+0000","last_peered":"2024-01-11T10:14:52.730006+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:52.730006+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":28288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":442}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.326221+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:22.326221+0000","last_peered":"2024-01-11T10:14:22.326221+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:22.326221+0000","last_undegraded":"2024-01-11T10:14:22.326221+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"44'478","reported_seq":498,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.083210+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:14:57.083210+0000","last_peered":"2024-01-11T10:14:57.083210+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:57.083210+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":479,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.171098+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:57.171098+0000","last_peered":"2024-01-11T10:14:57.171098+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:57.171098+0000","last_undegraded":"2024-01-11T10:14:57.171098+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"43'460","reported_seq":480,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.062532+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:14:57.062532+0000","last_peered":"2024-01-11T10:14:57.062532+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:57.062532+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":461,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285298+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:22.285298+0000","last_peered":"2024-01-11T10:14:22.285298+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:22.285298+0000","last_undegraded":"2024-01-11T10:14:22.285298+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"44'531","reported_seq":551,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.167121+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:14:57.167121+0000","last_peered":"2024-01-11T10:14:57.167121+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:57.167121+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.067209+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:57.067209+0000","last_peered":"2024-01-11T10:14:57.067209+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:57.067209+0000","last_undegraded":"2024-01-11T10:14:57.067209+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"41'415","reported_seq":429,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:52.955365+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:52.955365+0000","last_peered":"2024-01-11T10:14:52.955365+0000","last_clean":"2024-01-11T10:14:52.955365+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:52.955365+0000","last_undegraded":"2024-01-11T10:14:52.955365+0000","last_fullsized":"2024-01-11T10:14:52.955365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.061633+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:57.061633+0000","last_peered":"2024-01-11T10:14:57.061633+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:57.061633+0000","last_undegraded":"2024-01-11T10:14:57.061633+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"44'490","reported_seq":510,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.089192+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:14:57.089192+0000","last_peered":"2024-01-11T10:14:57.089192+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:57.089192+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":31424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.335164+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:22.335164+0000","last_peered":"2024-01-11T10:14:22.335164+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:22.335164+0000","last_undegraded":"2024-01-11T10:14:22.335164+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"41'461","reported_seq":473,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.248650+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:14:53.248650+0000","last_peered":"2024-01-11T10:14:53.248650+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:53.248650+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277632,"num_objects":462,"num_object_clones":0,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":462,"num_write_kb":29568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.177876+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:57.177876+0000","last_peered":"2024-01-11T10:14:57.177876+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:57.177876+0000","last_undegraded":"2024-01-11T10:14:57.177876+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":487129088,"num_objects":7433,"num_object_clones":0,"num_object_copies":22299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7433,"num_write_kb":475712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7417,"ondisk_log_size":7417,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112652,"kb_used_data":23364,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94259188,"statfs":{"total":96636764160,"available":96521408512,"internally_reserved":0,"allocated":23924736,"data_stored":376954082,"data_compressed":17895049,"data_compressed_allocated":23556096,"data_compressed_original":376815616,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124752,"kb_used_data":21576,"kb_used_omap":0,"kb_used_meta":103168,"kb_avail":94247088,"statfs":{"total":96636764160,"available":96509018112,"internally_reserved":0,"allocated":22093824,"data_stored":347403836,"data_compressed":16491666,"data_compressed_allocated":21708800,"data_compressed_original":347258880,"omap_allocated":0,"internal_metadata":105644032},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100716,"kb_used_data":19748,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94271124,"statfs":{"total":96636764160,"available":96533630976,"internally_reserved":0,"allocated":20221952,"data_stored":317846556,"data_compressed":15086313,"data_compressed_allocated":19857408,"data_compressed_original":317718528,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:58.771 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:14:58.771 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:14:58.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:14:58.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.940+0000 7f877266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:58.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:58.947+0000 7f877266d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:59.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 41 16 8206 8190 12.4837 10.9375 0.048439 0.0800332 2024-01-11T10:14:59.276 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:14:59.276 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:14:59.296 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-01-11T10:14:58.174663+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":487719456,"num_objects":7435,"num_object_clones":0,"num_object_copies":22305,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":7570,"num_write_kb":477606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7514,"ondisk_log_size":7514,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":413896,"kb_used_data":77544,"kb_used_omap":0,"kb_used_meta":336320,"kb_avail":377073464,"statfs":{"total":386547056640,"available":386123227136,"internally_reserved":0,"allocated":79405056,"data_stored":1248593506,"data_compressed":59270863,"data_compressed_allocated":78020608,"data_compressed_original":1248083968,"omap_allocated":0,"internal_metadata":344391680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27000832,"num_objects":412,"num_object_clones":0,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":412,"num_write_kb":26368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001228"},"pg_stats":[{"pgid":"5.f","version":"41'494","reported_seq":506,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.174688+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:14:53.174688+0000","last_peered":"2024-01-11T10:14:53.174688+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:14:53.174688+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":494,"log_dups_size":0,"ondisk_log_size":494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32440320,"num_objects":495,"num_object_clones":0,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":495,"num_write_kb":31680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":495}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"41'462","reported_seq":474,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.019587+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:14:53.019587+0000","last_peered":"2024-01-11T10:14:53.019587+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:14:53.019587+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30343168,"num_objects":463,"num_object_clones":0,"num_object_copies":1389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":463,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":463,"num_write_kb":29632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":463}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"41'431","reported_seq":443,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.151475+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:14:53.151475+0000","last_peered":"2024-01-11T10:14:53.151475+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:14:53.151475+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":432,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":432}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"43'465","reported_seq":485,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.066148+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:14:57.066148+0000","last_peered":"2024-01-11T10:14:57.066148+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:14:57.066148+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30539776,"num_objects":466,"num_object_clones":0,"num_object_copies":1398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":466,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":466,"num_write_kb":29824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":466}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"41'403","reported_seq":417,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:53.202119+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:14:53.202119+0000","last_peered":"2024-01-11T10:14:53.202119+0000","last_clean":"2024-01-11T10:14:53.202119+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:14:53.202119+0000","last_undegraded":"2024-01-11T10:14:53.202119+0000","last_fullsized":"2024-01-11T10:14:53.202119+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26476544,"num_objects":404,"num_object_clones":0,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":404,"num_write_kb":25856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"44'487","reported_seq":507,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.132428+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:14:57.132428+0000","last_peered":"2024-01-11T10:14:57.132428+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:14:57.132428+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":487,"log_dups_size":0,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31981568,"num_objects":488,"num_object_clones":0,"num_object_copies":1464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":488,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":488,"num_write_kb":31232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":488}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"44'517","reported_seq":533,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.175181+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:14:57.175181+0000","last_peered":"2024-01-11T10:14:57.175181+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:14:57.175181+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33947648,"num_objects":518,"num_object_clones":0,"num_object_copies":1554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":518,"num_write_kb":33152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":518}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"41'397","reported_seq":409,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:52.918962+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:14:52.918962+0000","last_peered":"2024-01-11T10:14:52.918962+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:14:52.918962+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26083328,"num_objects":398,"num_object_clones":0,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":398,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":398,"num_write_kb":25472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":398}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"41'485","reported_seq":497,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.197986+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:14:53.197986+0000","last_peered":"2024-01-11T10:14:53.197986+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:14:53.197986+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":485,"log_dups_size":0,"ondisk_log_size":485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31850496,"num_objects":486,"num_object_clones":0,"num_object_copies":1458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":486,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":486,"num_write_kb":31104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":486}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":203,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.077237+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:57.077237+0000","last_peered":"2024-01-11T10:14:57.077237+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:57.077237+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.127029+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:57.127029+0000","last_peered":"2024-01-11T10:14:57.127029+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:57.127029+0000","last_undegraded":"2024-01-11T10:14:57.127029+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"41'441","reported_seq":453,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:52.730006+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:14:52.730006+0000","last_peered":"2024-01-11T10:14:52.730006+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:14:52.730006+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28966912,"num_objects":442,"num_object_clones":0,"num_object_copies":1326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":442,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":442,"num_write_kb":28288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":442}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.326221+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:22.326221+0000","last_peered":"2024-01-11T10:14:22.326221+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:22.326221+0000","last_undegraded":"2024-01-11T10:14:22.326221+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"44'478","reported_seq":498,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.083210+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:14:57.083210+0000","last_peered":"2024-01-11T10:14:57.083210+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:14:57.083210+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31391744,"num_objects":479,"num_object_clones":0,"num_object_copies":1437,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":479,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":479,"num_write_kb":30656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":479}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.171098+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:57.171098+0000","last_peered":"2024-01-11T10:14:57.171098+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:57.171098+0000","last_undegraded":"2024-01-11T10:14:57.171098+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"43'460","reported_seq":480,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.062532+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:14:57.062532+0000","last_peered":"2024-01-11T10:14:57.062532+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:14:57.062532+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30212096,"num_objects":461,"num_object_clones":0,"num_object_copies":1383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":461,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":461,"num_write_kb":29504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":461}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":54,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.285298+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:22.285298+0000","last_peered":"2024-01-11T10:14:22.285298+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:22.285298+0000","last_undegraded":"2024-01-11T10:14:22.285298+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"44'531","reported_seq":551,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.167121+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:14:57.167121+0000","last_peered":"2024-01-11T10:14:57.167121+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:14:57.167121+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":531,"log_dups_size":0,"ondisk_log_size":531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34865152,"num_objects":532,"num_object_clones":0,"num_object_copies":1596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":532,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":532,"num_write_kb":34048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":532}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":35,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.067209+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:57.067209+0000","last_peered":"2024-01-11T10:14:57.067209+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:57.067209+0000","last_undegraded":"2024-01-11T10:14:57.067209+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"41'415","reported_seq":429,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-01-11T10:14:52.955365+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:14:52.955365+0000","last_peered":"2024-01-11T10:14:52.955365+0000","last_clean":"2024-01-11T10:14:52.955365+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:14:52.955365+0000","last_undegraded":"2024-01-11T10:14:52.955365+0000","last_fullsized":"2024-01-11T10:14:52.955365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27262976,"num_objects":416,"num_object_clones":0,"num_object_copies":1248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":416,"num_write_kb":26624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":72,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.061633+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:57.061633+0000","last_peered":"2024-01-11T10:14:57.061633+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:57.061633+0000","last_undegraded":"2024-01-11T10:14:57.061633+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"44'490","reported_seq":510,"reported_epoch":44,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:57.089192+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:14:57.089192+0000","last_peered":"2024-01-11T10:14:57.089192+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:14:57.089192+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":32178176,"num_objects":491,"num_object_clones":0,"num_object_copies":1473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":491,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":491,"num_write_kb":31424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":491}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":23,"reported_epoch":41,"state":"active+undersized","last_fresh":"2024-01-11T10:14:22.335164+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:22.335164+0000","last_peered":"2024-01-11T10:14:22.335164+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:22.335164+0000","last_undegraded":"2024-01-11T10:14:22.335164+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"41'461","reported_seq":473,"reported_epoch":41,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:53.248650+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:14:53.248650+0000","last_peered":"2024-01-11T10:14:53.248650+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:14:53.248650+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30277632,"num_objects":462,"num_object_clones":0,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":462,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":462,"num_write_kb":29568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":462}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":60,"reported_epoch":44,"state":"active+undersized","last_fresh":"2024-01-11T10:14:57.177876+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:57.177876+0000","last_peered":"2024-01-11T10:14:57.177876+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:57.177876+0000","last_undegraded":"2024-01-11T10:14:57.177876+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":487129088,"num_objects":7433,"num_object_clones":0,"num_object_copies":22299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7433,"num_write_kb":475712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7417,"ondisk_log_size":7417,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706059,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":112652,"kb_used_data":23364,"kb_used_omap":0,"kb_used_meta":89280,"kb_avail":94259188,"statfs":{"total":96636764160,"available":96521408512,"internally_reserved":0,"allocated":23924736,"data_stored":376954082,"data_compressed":17895049,"data_compressed_allocated":23556096,"data_compressed_original":376815616,"omap_allocated":0,"internal_metadata":91422720},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":124752,"kb_used_data":21576,"kb_used_omap":0,"kb_used_meta":103168,"kb_avail":94247088,"statfs":{"total":96636764160,"available":96509018112,"internally_reserved":0,"allocated":22093824,"data_stored":347403836,"data_compressed":16491666,"data_compressed_allocated":21708800,"data_compressed_original":347258880,"omap_allocated":0,"internal_metadata":105644032},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":100716,"kb_used_data":19748,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94271124,"statfs":{"total":96636764160,"available":96533630976,"internally_reserved":0,"allocated":20221952,"data_stored":317846556,"data_compressed":15086313,"data_compressed_allocated":19857408,"data_compressed_original":317718528,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:14:59.297 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 5.c 5.4 1.0 5.7 5.0 2024-01-11T10:14:59.422 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:59.429+0000 7fca2cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:59.426 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:14:59.443+0000 7fca2cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:14:59.627 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.0,5.4,5.7] on osd.0 to force-backfill; instructing pg(s) [1.0,5.c] on osd.3 to force-backfill; 2024-01-11T10:14:59.627 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:15:00.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 42 16 8387 8371 12.4558 11.3125 0.158177 0.0801476 2024-01-11T10:15:00.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:00.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:00.867+0000 7f447462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:00.854 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:00.872+0000 7f447462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:00.970 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:00.997 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:01.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 43 16 8585 8569 12.4539 12.375 0.0202968 0.0802278 2024-01-11T10:15:02.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 44 16 8783 8767 12.4521 12.375 0.0583876 0.0802098 2024-01-11T10:15:02.998 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:03.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 45 16 8990 8974 12.4629 12.9375 0.115318 0.0801381 2024-01-11T10:15:03.118 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:03.134+0000 7fa3e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:03.122 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:03.139+0000 7fa3e586d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:03.205 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:03.224 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:03.224 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:03.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~160s 2024-01-11T10:15:04.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 46 16 9181 9165 12.4514 11.9375 0.0671949 0.0802156 2024-01-11T10:15:04.642 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:04.642 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:04.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:04.780 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:04.794+0000 7effc1c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:04.783 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:04.801+0000 7effc1c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:04.988 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:04.988 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:05.001 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-01-11T10:15:04.175945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":585695776,"num_objects":8930,"num_object_clones":0,"num_object_copies":26790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7898,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":9065,"num_write_kb":573286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9009,"ondisk_log_size":9009,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":469416,"kb_used_data":88648,"kb_used_omap":0,"kb_used_meta":380736,"kb_avail":377017944,"statfs":{"total":386547056640,"available":386066374656,"internally_reserved":0,"allocated":90775552,"data_stored":1429498826,"data_compressed":67859983,"data_compressed_allocated":89325568,"data_compressed_original":1428963328,"omap_allocated":0,"internal_metadata":389873664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":124977152,"num_objects":1907,"num_object_clones":0,"num_object_copies":5721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1695,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1907,"num_write_kb":122048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001879"},"pg_stats":[{"pgid":"5.f","version":"45'616","reported_seq":634,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.180659+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:03.180659+0000","last_peered":"2024-01-11T10:15:03.180659+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:03.180659+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"45'561","reported_seq":579,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.035987+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:03.035987+0000","last_peered":"2024-01-11T10:15:03.035987+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:03.035987+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":561,"log_dups_size":0,"ondisk_log_size":561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36831232,"num_objects":562,"num_object_clones":0,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":562,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":562,"num_write_kb":35968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":562}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"45'571","reported_seq":589,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.155536+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:03.155536+0000","last_peered":"2024-01-11T10:15:03.155536+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:03.155536+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":571,"log_dups_size":0,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37486592,"num_objects":572,"num_object_clones":0,"num_object_copies":1716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":572,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":572,"num_write_kb":36608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":572}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"45'529","reported_seq":551,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.127433+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:02.127433+0000","last_peered":"2024-01-11T10:15:02.127433+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:02.127433+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34734080,"num_objects":530,"num_object_clones":0,"num_object_copies":1590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":530,"num_write_kb":33920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":530}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"45'516","reported_seq":536,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:03.115565+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:03.115565+0000","last_peered":"2024-01-11T10:15:03.115565+0000","last_clean":"2024-01-11T10:15:03.115565+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:03.115565+0000","last_undegraded":"2024-01-11T10:15:03.115565+0000","last_fullsized":"2024-01-11T10:15:03.115565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33882112,"num_objects":517,"num_object_clones":0,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":517,"num_write_kb":33088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"45'555","reported_seq":577,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.132608+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:02.132608+0000","last_peered":"2024-01-11T10:15:02.132608+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:02.132608+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":555,"log_dups_size":0,"ondisk_log_size":555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36438016,"num_objects":556,"num_object_clones":0,"num_object_copies":1668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":556,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":556,"num_write_kb":35584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":556}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"45'588","reported_seq":606,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.072821+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:02.072821+0000","last_peered":"2024-01-11T10:15:02.072821+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:02.072821+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38600704,"num_objects":589,"num_object_clones":0,"num_object_copies":1767,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":589,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":589,"num_write_kb":37696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":589}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"45'514","reported_seq":532,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.084340+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:03.084340+0000","last_peered":"2024-01-11T10:15:03.084340+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:03.084340+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":515}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"45'616","reported_seq":634,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.177788+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:03.177788+0000","last_peered":"2024-01-11T10:15:03.177788+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:03.177788+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40435712,"num_objects":617,"num_object_clones":0,"num_object_copies":1851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":617,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":617,"num_write_kb":39488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":617}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":205,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:58.228718+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:58.228718+0000","last_peered":"2024-01-11T10:14:58.228718+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:58.228718+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.101706+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:58.101706+0000","last_peered":"2024-01-11T10:14:58.101706+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:58.101706+0000","last_undegraded":"2024-01-11T10:14:58.101706+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"45'563","reported_seq":581,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.022384+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:03.022384+0000","last_peered":"2024-01-11T10:15:03.022384+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:03.022384+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36962304,"num_objects":564,"num_object_clones":0,"num_object_copies":1692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":564,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":564,"num_write_kb":36096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":564}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.320325+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:58.320325+0000","last_peered":"2024-01-11T10:14:58.320325+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:58.320325+0000","last_undegraded":"2024-01-11T10:14:58.320325+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"45'525","reported_seq":547,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.037485+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:02.037485+0000","last_peered":"2024-01-11T10:15:02.037485+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:02.037485+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34471936,"num_objects":526,"num_object_clones":0,"num_object_copies":1578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":526,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":526,"num_write_kb":33664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":526}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.114342+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:58.114342+0000","last_peered":"2024-01-11T10:14:58.114342+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:58.114342+0000","last_undegraded":"2024-01-11T10:14:58.114342+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"45'511","reported_seq":533,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.167174+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:02.167174+0000","last_peered":"2024-01-11T10:15:02.167174+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:02.167174+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":512,"num_object_clones":0,"num_object_copies":1536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":512,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":512,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":512}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.332897+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:58.332897+0000","last_peered":"2024-01-11T10:14:58.332897+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:58.332897+0000","last_undegraded":"2024-01-11T10:14:58.332897+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"45'595","reported_seq":617,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.168545+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:02.168545+0000","last_peered":"2024-01-11T10:15:02.168545+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:02.168545+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39059456,"num_objects":596,"num_object_clones":0,"num_object_copies":1788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":596,"num_write_kb":38144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":596}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.236679+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:58.236679+0000","last_peered":"2024-01-11T10:14:58.236679+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:58.236679+0000","last_undegraded":"2024-01-11T10:14:58.236679+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"45'514","reported_seq":534,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:03.034316+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:03.034316+0000","last_peered":"2024-01-11T10:15:03.034316+0000","last_clean":"2024-01-11T10:15:03.034316+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:03.034316+0000","last_undegraded":"2024-01-11T10:15:03.034316+0000","last_fullsized":"2024-01-11T10:15:03.034316+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":514,"log_dups_size":0,"ondisk_log_size":514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33751040,"num_objects":515,"num_object_clones":0,"num_object_copies":1545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":515,"num_write_kb":32960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.247746+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:58.247746+0000","last_peered":"2024-01-11T10:14:58.247746+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:58.247746+0000","last_undegraded":"2024-01-11T10:14:58.247746+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"45'541","reported_seq":563,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:02.053288+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:02.053288+0000","last_peered":"2024-01-11T10:15:02.053288+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:02.053288+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35520512,"num_objects":542,"num_object_clones":0,"num_object_copies":1626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":542,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":542,"num_write_kb":34688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":542}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.325681+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:58.325681+0000","last_peered":"2024-01-11T10:14:58.325681+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:58.325681+0000","last_undegraded":"2024-01-11T10:14:58.325681+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"45'597","reported_seq":615,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:03.108559+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:03.108559+0000","last_peered":"2024-01-11T10:15:03.108559+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:03.108559+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39190528,"num_objects":598,"num_object_clones":0,"num_object_copies":1794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":598,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":598,"num_write_kb":38272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":598}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.108498+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:58.108498+0000","last_peered":"2024-01-11T10:14:58.108498+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:58.108498+0000","last_undegraded":"2024-01-11T10:14:58.108498+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":585105408,"num_objects":8928,"num_object_clones":0,"num_object_copies":26784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7896,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8928,"num_write_kb":571392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8912,"ondisk_log_size":8912,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141120,"kb_used_data":29624,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94230720,"statfs":{"total":96636764160,"available":96492257280,"internally_reserved":0,"allocated":30334976,"data_stored":479006614,"data_compressed":22740433,"data_compressed_allocated":29933568,"data_compressed_original":478855168,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":131384,"kb_used_data":24048,"kb_used_omap":0,"kb_used_meta":107328,"kb_avail":94240456,"statfs":{"total":96636764160,"available":96502226944,"internally_reserved":0,"allocated":24625152,"data_stored":387649430,"data_compressed":18402434,"data_compressed_allocated":24223744,"data_compressed_original":387497984,"omap_allocated":0,"internal_metadata":109903872},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":121136,"kb_used_data":22120,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94250704,"statfs":{"total":96636764160,"available":96512720896,"internally_reserved":0,"allocated":22650880,"data_stored":356453750,"data_compressed":16919281,"data_compressed_allocated":22269952,"data_compressed_original":356319232,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:05.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 5.e 5.d 5.8 5.4 1.0 5.6 2024-01-11T10:15:05.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 47 16 9385 9369 12.4577 12.75 0.105516 0.0801861 2024-01-11T10:15:05.129 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:05.132+0000 7fa4f7f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:05.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:05.150+0000 7fa4f7f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:05.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:05.335 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.4,5.8,5.d,5.e] on osd.0 to force-backfill; instructing pg(s) [1.0,5.6] on osd.3 to force-backfill; 2024-01-11T10:15:05.335 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:15:05.351 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:05.367+0000 7efbfbe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:05.355 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:05.372+0000 7efbfbe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:05.447 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:05.470 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:06.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 48 16 9581 9565 12.4534 12.25 0.032307 0.0802329 2024-01-11T10:15:07.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 49 16 9800 9784 12.4785 13.6875 0.0130856 0.0800289 2024-01-11T10:15:07.471 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:07.592 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:07.608+0000 7f3651308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:07.596 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:07.614+0000 7f3651308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:07.709 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:07.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:08.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 50 16 10004 9988 12.484 12.75 0.0902721 0.0800449 2024-01-11T10:15:08.775 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~165s 2024-01-11T10:15:09.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 51 16 10213 10197 12.4953 13.0625 0.092074 0.0799499 2024-01-11T10:15:09.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:09.849 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:09.864+0000 7f7736b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:09.852 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:09.870+0000 7f7736b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:09.981 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:10.001 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:10.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 52 16 10413 10397 12.4953 12.5 0.0168681 0.0799707 2024-01-11T10:15:10.355 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:10.355 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:10.355 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:15:10.355 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:15:10.491 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:10.503+0000 7fe29d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.496 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:10.513+0000 7fe29d26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:10.577 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:10.581 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:10.581 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:15:10.707 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:10.720+0000 7f2c1759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.711 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:10.728+0000 7f2c1759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:10.726 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:10.727 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:10.730 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:10.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:15:10.855 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:10.867+0000 7fcced19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.859 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:10.876+0000 7fcced19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.864 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:10.868 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:10.868 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:15:10.988 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:11.000+0000 7f190ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:10.991 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:11.009+0000 7f190ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:11.015 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:11.016 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:11.019 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:11.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 53 16 10605 10589 12.486 12 0.0690921 0.0799938 2024-01-11T10:15:12.001 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:12.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 54 16 10818 10802 12.5013 13.3125 0.0114746 0.079905 2024-01-11T10:15:12.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:12.147+0000 7fbfc20f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:12.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:12.152+0000 7fbfc20f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:12.230 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:12.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:12.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:13.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 55 16 10999 10983 12.4796 11.3125 0.0396872 0.0800337 2024-01-11T10:15:14.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 56 16 11195 11179 12.4755 12.25 0.0694895 0.0800515 2024-01-11T10:15:14.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:14.280 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~171s 2024-01-11T10:15:14.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:14.405+0000 7fa50939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:14.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:14.411+0000 7fa50939e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:14.511 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:14.536 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:15.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 57 16 11404 11388 12.4858 13.0625 0.101467 0.080017 2024-01-11T10:15:16.020 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:16.020 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:16.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:16.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 58 16 11587 11571 12.4677 11.4375 0.104473 0.0801325 2024-01-11T10:15:16.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:16.154+0000 7f6a3262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:16.145 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:16.163+0000 7f6a3262d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:16.393 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:16.393 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:16.408 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-01-11T10:15:16.179174+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":717226528,"num_objects":10937,"num_object_clones":0,"num_object_copies":32811,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9673,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":11072,"num_write_kb":701734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11016,"ondisk_log_size":11016,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":490684,"kb_used_data":105756,"kb_used_omap":0,"kb_used_meta":384896,"kb_avail":376996676,"statfs":{"total":386547056640,"available":386044596224,"internally_reserved":0,"allocated":108294144,"data_stored":1709796298,"data_compressed":81170007,"data_compressed_allocated":106844160,"data_compressed_original":1709260800,"omap_allocated":0,"internal_metadata":394133504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":131530752,"num_objects":2007,"num_object_clones":0,"num_object_copies":6021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2007,"num_write_kb":128448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003229"},"pg_stats":[{"pgid":"5.f","version":"45'743","reported_seq":761,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.117971+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:13.117971+0000","last_peered":"2024-01-11T10:15:13.117971+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:13.117971+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48758784,"num_objects":744,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":744,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":744,"num_write_kb":47616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":744}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"45'691","reported_seq":709,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.076068+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:13.076068+0000","last_peered":"2024-01-11T10:15:13.076068+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:13.076068+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":691,"log_dups_size":0,"ondisk_log_size":691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45350912,"num_objects":692,"num_object_clones":0,"num_object_copies":2076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":692,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":692,"num_write_kb":44288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":692}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"45'719","reported_seq":737,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.014190+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:13.014190+0000","last_peered":"2024-01-11T10:15:13.014190+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:13.014190+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":719,"log_dups_size":0,"ondisk_log_size":719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47185920,"num_objects":720,"num_object_clones":0,"num_object_copies":2160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":720,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":720,"num_write_kb":46080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":720}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"45'649","reported_seq":671,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:12.153316+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:12.153316+0000","last_peered":"2024-01-11T10:15:12.153316+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:12.153316+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":649,"log_dups_size":0,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42598400,"num_objects":650,"num_object_clones":0,"num_object_copies":1950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":650,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":650,"num_write_kb":41600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":650}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"45'630","reported_seq":650,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:13.094127+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:13.094127+0000","last_peered":"2024-01-11T10:15:13.094127+0000","last_clean":"2024-01-11T10:15:13.094127+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:13.094127+0000","last_undegraded":"2024-01-11T10:15:13.094127+0000","last_fullsized":"2024-01-11T10:15:13.094127+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":630,"log_dups_size":0,"ondisk_log_size":630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41353216,"num_objects":631,"num_object_clones":0,"num_object_copies":1893,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":631,"num_write_kb":40384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"45'684","reported_seq":706,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:12.104583+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:12.104583+0000","last_peered":"2024-01-11T10:15:12.104583+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:12.104583+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":685,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"45'713","reported_seq":731,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:11.818986+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:11.818986+0000","last_peered":"2024-01-11T10:15:11.818986+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:11.818986+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46792704,"num_objects":714,"num_object_clones":0,"num_object_copies":2142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":714,"num_write_kb":45696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":714}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"45'625","reported_seq":643,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.136515+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:13.136515+0000","last_peered":"2024-01-11T10:15:13.136515+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:13.136515+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41025536,"num_objects":626,"num_object_clones":0,"num_object_copies":1878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":626,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":626,"num_write_kb":40064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":626}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"45'732","reported_seq":750,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.123663+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:13.123663+0000","last_peered":"2024-01-11T10:15:13.123663+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:13.123663+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48037888,"num_objects":733,"num_object_clones":0,"num_object_copies":2199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":733,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":733,"num_write_kb":46912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":733}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":205,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:58.228718+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:58.228718+0000","last_peered":"2024-01-11T10:14:58.228718+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:58.228718+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.101706+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:58.101706+0000","last_peered":"2024-01-11T10:14:58.101706+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:58.101706+0000","last_undegraded":"2024-01-11T10:14:58.101706+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"45'684","reported_seq":702,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.088262+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:13.088262+0000","last_peered":"2024-01-11T10:15:13.088262+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:13.088262+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":685,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":685}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.320325+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:58.320325+0000","last_peered":"2024-01-11T10:14:58.320325+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:58.320325+0000","last_undegraded":"2024-01-11T10:14:58.320325+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"45'643","reported_seq":665,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:12.012483+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:12.012483+0000","last_peered":"2024-01-11T10:15:12.012483+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:12.012483+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42205184,"num_objects":644,"num_object_clones":0,"num_object_copies":1932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":644,"num_write_kb":41216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":644}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.114342+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:58.114342+0000","last_peered":"2024-01-11T10:14:58.114342+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:58.114342+0000","last_undegraded":"2024-01-11T10:14:58.114342+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"45'648","reported_seq":670,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:12.048918+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:12.048918+0000","last_peered":"2024-01-11T10:15:12.048918+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:12.048918+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42532864,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":649,"num_write_kb":41536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":649}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.332897+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:58.332897+0000","last_peered":"2024-01-11T10:14:58.332897+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:58.332897+0000","last_undegraded":"2024-01-11T10:14:58.332897+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"45'735","reported_seq":757,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:12.070967+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:12.070967+0000","last_peered":"2024-01-11T10:15:12.070967+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:12.070967+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":735,"log_dups_size":0,"ondisk_log_size":735,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48234496,"num_objects":736,"num_object_clones":0,"num_object_copies":2208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":736,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":736,"num_write_kb":47104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":736}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.236679+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:58.236679+0000","last_peered":"2024-01-11T10:14:58.236679+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:58.236679+0000","last_undegraded":"2024-01-11T10:14:58.236679+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"45'632","reported_seq":652,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:13.029001+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:13.029001+0000","last_peered":"2024-01-11T10:15:13.029001+0000","last_clean":"2024-01-11T10:15:13.029001+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:13.029001+0000","last_undegraded":"2024-01-11T10:15:13.029001+0000","last_fullsized":"2024-01-11T10:15:13.029001+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41484288,"num_objects":633,"num_object_clones":0,"num_object_copies":1899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":633,"num_write_kb":40512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.247746+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:58.247746+0000","last_peered":"2024-01-11T10:14:58.247746+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:58.247746+0000","last_undegraded":"2024-01-11T10:14:58.247746+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"45'680","reported_seq":702,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:12.094219+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:12.094219+0000","last_peered":"2024-01-11T10:15:12.094219+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:12.094219+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":680,"log_dups_size":0,"ondisk_log_size":680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44630016,"num_objects":681,"num_object_clones":0,"num_object_copies":2043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":681,"num_write_kb":43584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":681}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.325681+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:58.325681+0000","last_peered":"2024-01-11T10:14:58.325681+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:58.325681+0000","last_undegraded":"2024-01-11T10:14:58.325681+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"45'711","reported_seq":729,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:13.206290+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:13.206290+0000","last_peered":"2024-01-11T10:15:13.206290+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:13.206290+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46661632,"num_objects":712,"num_object_clones":0,"num_object_copies":2136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":712,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":712,"num_write_kb":45568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":712}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.108498+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:58.108498+0000","last_peered":"2024-01-11T10:14:58.108498+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:58.108498+0000","last_undegraded":"2024-01-11T10:14:58.108498+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":716636160,"num_objects":10935,"num_object_clones":0,"num_object_copies":32805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9671,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":699840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10919,"ondisk_log_size":10919,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706079,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":139228,"kb_used_data":36052,"kb_used_omap":0,"kb_used_meta":103168,"kb_avail":94232612,"statfs":{"total":96636764160,"available":96494194688,"internally_reserved":0,"allocated":36917248,"data_stored":584322966,"data_compressed":27741417,"data_compressed_allocated":36515840,"data_compressed_original":584171520,"omap_allocated":0,"internal_metadata":105644032},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":141300,"kb_used_data":29804,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94230540,"statfs":{"total":96636764160,"available":96492072960,"internally_reserved":0,"allocated":30519296,"data_stored":481955734,"data_compressed":22880602,"data_compressed_allocated":30117888,"data_compressed_original":481804288,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706078,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134380,"kb_used_data":27044,"kb_used_omap":0,"kb_used_meta":107328,"kb_avail":94237460,"statfs":{"total":96636764160,"available":96499159040,"internally_reserved":0,"allocated":27693056,"data_stored":437128566,"data_compressed":20750153,"data_compressed_allocated":27312128,"data_compressed_original":436994048,"omap_allocated":0,"internal_metadata":109903872},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:16.537 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:16.660 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:16.677+0000 7fd69f11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:16.663 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:16.682+0000 7fd69f11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:16.822 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:16.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:17.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 59 16 11784 11768 12.4651 12.3125 0.160893 0.0801314 2024-01-11T10:15:18.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:15:18.132031+0000 min lat: 0.00835493 max lat: 0.30334 avg lat: 0.0802386 2024-01-11T10:15:18.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:15:18.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 60 16 11969 11953 12.45 11.5625 0.111017 0.0802386 2024-01-11T10:15:18.841 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:18.979 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:18.996+0000 7f1aaca17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:18.982 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:19.001+0000 7f1aaca17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:19.091 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:19.110 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:19.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 61 16 12167 12151 12.4488 12.375 0.106858 0.0802722 2024-01-11T10:15:19.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~176s 2024-01-11T10:15:20.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 62 16 12355 12339 12.4375 11.75 0.115272 0.0803144 2024-01-11T10:15:21.111 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:21.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 63 16 12568 12552 12.4513 13.3125 0.0149454 0.0802573 2024-01-11T10:15:21.241 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:21.258+0000 7f113602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:21.244 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:21.263+0000 7f113602d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:21.331 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:21.349 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:21.410 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:21.410 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:21.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:21.534 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:21.549+0000 7f929079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:21.538 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:21.557+0000 7f929079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:21.745 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:21.745 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:21.758 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-01-11T10:15:20.180156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":781713952,"num_objects":11921,"num_object_clones":0,"num_object_copies":35763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":12056,"num_write_kb":764710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12000,"ondisk_log_size":12000,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":490632,"kb_used_data":114024,"kb_used_omap":0,"kb_used_meta":376576,"kb_avail":376996728,"statfs":{"total":386547056640,"available":386044649472,"internally_reserved":0,"allocated":116760576,"data_stored":1845259210,"data_compressed":87602509,"data_compressed_allocated":115310592,"data_compressed_original":1844723712,"omap_allocated":0,"internal_metadata":385613824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":166854656,"num_objects":2546,"num_object_clones":0,"num_object_copies":7638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2546,"num_write_kb":162944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003083"},"pg_stats":[{"pgid":"5.f","version":"45'800","reported_seq":818,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.995173+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:17.995173+0000","last_peered":"2024-01-11T10:15:17.995173+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:17.995173+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":800,"log_dups_size":0,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52494336,"num_objects":801,"num_object_clones":0,"num_object_copies":2403,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":801,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":801,"num_write_kb":51264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":801}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"45'745","reported_seq":763,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:18.225989+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:18.225989+0000","last_peered":"2024-01-11T10:15:18.225989+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:18.225989+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48889856,"num_objects":746,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":47744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":746}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"45'786","reported_seq":804,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:18.213287+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:18.213287+0000","last_peered":"2024-01-11T10:15:18.213287+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:18.213287+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":786,"log_dups_size":0,"ondisk_log_size":786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51576832,"num_objects":787,"num_object_clones":0,"num_object_copies":2361,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":787,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":787,"num_write_kb":50368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":787}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"45'714","reported_seq":736,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:16.927966+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:16.927966+0000","last_peered":"2024-01-11T10:15:16.927966+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:16.927966+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46858240,"num_objects":715,"num_object_clones":0,"num_object_copies":2145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":715,"num_write_kb":45760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":715}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"45'684","reported_seq":704,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:18.083334+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:18.083334+0000","last_peered":"2024-01-11T10:15:18.083334+0000","last_clean":"2024-01-11T10:15:18.083334+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:18.083334+0000","last_undegraded":"2024-01-11T10:15:18.083334+0000","last_fullsized":"2024-01-11T10:15:18.083334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":685,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"45'744","reported_seq":766,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.153128+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:17.153128+0000","last_peered":"2024-01-11T10:15:17.153128+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:17.153128+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":745,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":745}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"45'787","reported_seq":805,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.177150+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:17.177150+0000","last_peered":"2024-01-11T10:15:17.177150+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:17.177150+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":788}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"45'688","reported_seq":706,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.994679+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:17.994679+0000","last_peered":"2024-01-11T10:15:17.994679+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:17.994679+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":688,"log_dups_size":0,"ondisk_log_size":688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154304,"num_objects":689,"num_object_clones":0,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":689,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":689,"num_write_kb":44096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":689}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"45'804","reported_seq":822,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:18.221202+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:18.221202+0000","last_peered":"2024-01-11T10:15:18.221202+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:18.221202+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":805,"num_write_kb":51520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":805}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":205,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:58.228718+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:58.228718+0000","last_peered":"2024-01-11T10:14:58.228718+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:58.228718+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.101706+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:58.101706+0000","last_peered":"2024-01-11T10:14:58.101706+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:58.101706+0000","last_undegraded":"2024-01-11T10:14:58.101706+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"45'736","reported_seq":754,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:18.041384+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:18.041384+0000","last_peered":"2024-01-11T10:15:18.041384+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:18.041384+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48300032,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":737,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.320325+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:58.320325+0000","last_peered":"2024-01-11T10:14:58.320325+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:58.320325+0000","last_undegraded":"2024-01-11T10:14:58.320325+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"45'718","reported_seq":740,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.066719+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:17.066719+0000","last_peered":"2024-01-11T10:15:17.066719+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:17.066719+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47120384,"num_objects":719,"num_object_clones":0,"num_object_copies":2157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":719,"num_write_kb":46016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":719}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.114342+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:58.114342+0000","last_peered":"2024-01-11T10:14:58.114342+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:58.114342+0000","last_undegraded":"2024-01-11T10:14:58.114342+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"45'698","reported_seq":720,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.066426+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:17.066426+0000","last_peered":"2024-01-11T10:15:17.066426+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:17.066426+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45809664,"num_objects":699,"num_object_clones":0,"num_object_copies":2097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":699,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":699,"num_write_kb":44736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":699}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.332897+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:58.332897+0000","last_peered":"2024-01-11T10:14:58.332897+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:58.332897+0000","last_undegraded":"2024-01-11T10:14:58.332897+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"45'804","reported_seq":826,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:17.009751+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:17.009751+0000","last_peered":"2024-01-11T10:15:17.009751+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:17.009751+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":804,"log_dups_size":0,"ondisk_log_size":804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52756480,"num_objects":805,"num_object_clones":0,"num_object_copies":2415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":805,"num_write_kb":51520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.236679+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:58.236679+0000","last_peered":"2024-01-11T10:14:58.236679+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:58.236679+0000","last_undegraded":"2024-01-11T10:14:58.236679+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"45'684","reported_seq":704,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:18.088895+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:18.088895+0000","last_peered":"2024-01-11T10:15:18.088895+0000","last_clean":"2024-01-11T10:15:18.088895+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:18.088895+0000","last_undegraded":"2024-01-11T10:15:18.088895+0000","last_fullsized":"2024-01-11T10:15:18.088895+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":684,"log_dups_size":0,"ondisk_log_size":684,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":685,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.247746+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:58.247746+0000","last_peered":"2024-01-11T10:14:58.247746+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:58.247746+0000","last_undegraded":"2024-01-11T10:14:58.247746+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"45'742","reported_seq":764,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:16.949143+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:16.949143+0000","last_peered":"2024-01-11T10:15:16.949143+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:16.949143+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693248,"num_objects":743,"num_object_clones":0,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":743,"num_write_kb":47552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":743}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.325681+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:58.325681+0000","last_peered":"2024-01-11T10:14:58.325681+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:58.325681+0000","last_undegraded":"2024-01-11T10:14:58.325681+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"45'769","reported_seq":787,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:18.167202+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:18.167202+0000","last_peered":"2024-01-11T10:15:18.167202+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:18.167202+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50462720,"num_objects":770,"num_object_clones":0,"num_object_copies":2310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":770,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":770,"num_write_kb":49280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":770}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.108498+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:58.108498+0000","last_peered":"2024-01-11T10:14:58.108498+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:58.108498+0000","last_undegraded":"2024-01-11T10:14:58.108498+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":781123584,"num_objects":11919,"num_object_clones":0,"num_object_copies":35757,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10549,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11919,"num_write_kb":762816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11903,"ondisk_log_size":11903,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706084,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":146572,"kb_used_data":39236,"kb_used_omap":0,"kb_used_meta":107328,"kb_avail":94225268,"statfs":{"total":96636764160,"available":96486674432,"internally_reserved":0,"allocated":40177664,"data_stored":636489622,"data_compressed":30218569,"data_compressed_allocated":39776256,"data_compressed_original":636338176,"omap_allocated":0,"internal_metadata":109903872},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":131480,"kb_used_data":32464,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94240360,"statfs":{"total":96636764160,"available":96502128640,"internally_reserved":0,"allocated":33243136,"data_stored":525537174,"data_compressed":24950081,"data_compressed_allocated":32841728,"data_compressed_original":525385728,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":136804,"kb_used_data":29468,"kb_used_omap":0,"kb_used_meta":107328,"kb_avail":94235036,"statfs":{"total":96636764160,"available":96496676864,"internally_reserved":0,"allocated":30175232,"data_stored":476843382,"data_compressed":22636024,"data_compressed_allocated":29794304,"data_compressed_original":476708864,"omap_allocated":0,"internal_metadata":109903872},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:21.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 5.e 5.d 5.c 5.a 5.9 5.8 1.0 5.7 5.6 5.5 5.1 2024-01-11T10:15:21.888 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:21.890+0000 7f9db239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:21.891 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:21.910+0000 7f9db239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:22.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 64 16 12733 12717 12.4179 10.3125 0.208444 0.0804307 2024-01-11T10:15:22.256 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.7,5.8,5.d,5.e] on osd.0 to force-recovery; instructing pg(s) [5.9] on osd.1 to force-recovery; instructing pg(s) [1.0,5.1,5.5,5.6,5.a,5.c] on osd.3 to force-recovery; 2024-01-11T10:15:22.256 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:15:23.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 65 16 12909 12893 12.3961 11 0.103513 0.080591 2024-01-11T10:15:23.350 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:23.467 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:23.484+0000 7fad35f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:23.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:23.489+0000 7fad35f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:23.599 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:23.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:24.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 66 16 13107 13091 12.3958 12.375 0.0842709 0.0806068 2024-01-11T10:15:25.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 67 16 13298 13282 12.3889 11.9375 0.149419 0.0806106 2024-01-11T10:15:25.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~182s 2024-01-11T10:15:25.620 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:25.738 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:25.755+0000 7fc6e702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:25.742 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:25.761+0000 7fc6e702d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:25.847 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:25.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:26.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 68 16 13477 13461 12.3712 11.1875 0.012976 0.0807593 2024-01-11T10:15:27.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 69 16 13657 13641 12.355 11.25 0.11688 0.0808551 2024-01-11T10:15:27.276 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:27.276 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:27.276 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:27.393 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:27.408+0000 7fc7d146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:27.396 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:27.415+0000 7fc7d146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:27.670 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:27.687 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-01-11T10:15:26.181826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":843121184,"num_objects":12858,"num_object_clones":0,"num_object_copies":38574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11369,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":12993,"num_write_kb":824678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12937,"ondisk_log_size":12937,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":511132,"kb_used_data":122044,"kb_used_omap":0,"kb_used_meta":389056,"kb_avail":376976228,"statfs":{"total":386547056640,"available":386023657472,"internally_reserved":0,"allocated":124973056,"data_stored":1976658890,"data_compressed":93842069,"data_compressed_allocated":123523072,"data_compressed_original":1976123392,"omap_allocated":0,"internal_metadata":398393344},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":125894656,"num_objects":1921,"num_object_clones":0,"num_object_copies":5763,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1921,"num_write_kb":122944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003234"},"pg_stats":[{"pgid":"5.f","version":"45'862","reported_seq":880,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:23.227591+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:23.227591+0000","last_peered":"2024-01-11T10:15:23.227591+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:23.227591+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56557568,"num_objects":863,"num_object_clones":0,"num_object_copies":2589,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":863,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":863,"num_write_kb":55232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":863}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"45'823","reported_seq":841,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:23.204240+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:23.204240+0000","last_peered":"2024-01-11T10:15:23.204240+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:23.204240+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":823,"log_dups_size":0,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54001664,"num_objects":824,"num_object_clones":0,"num_object_copies":2472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":824,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":824,"num_write_kb":52736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":824}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"45'859","reported_seq":877,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:23.075745+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:23.075745+0000","last_peered":"2024-01-11T10:15:23.075745+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:23.075745+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":859,"log_dups_size":0,"ondisk_log_size":859,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56360960,"num_objects":860,"num_object_clones":0,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":860,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":860,"num_write_kb":55040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":860}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"45'771","reported_seq":793,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:21.805285+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:21.805285+0000","last_peered":"2024-01-11T10:15:21.805285+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:21.805285+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":771,"log_dups_size":0,"ondisk_log_size":771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50593792,"num_objects":772,"num_object_clones":0,"num_object_copies":2316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":772,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":772,"num_write_kb":49408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":772}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"45'744","reported_seq":764,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:22.981498+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:22.981498+0000","last_peered":"2024-01-11T10:15:22.981498+0000","last_clean":"2024-01-11T10:15:22.981498+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:22.981498+0000","last_undegraded":"2024-01-11T10:15:22.981498+0000","last_fullsized":"2024-01-11T10:15:22.981498+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"45'799","reported_seq":821,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:21.888874+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:21.888874+0000","last_peered":"2024-01-11T10:15:21.888874+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:21.888874+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":799,"log_dups_size":0,"ondisk_log_size":799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52428800,"num_objects":800,"num_object_clones":0,"num_object_copies":2400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":800,"num_write_kb":51200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":800}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"45'829","reported_seq":847,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:22.036074+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:22.036074+0000","last_peered":"2024-01-11T10:15:22.036074+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:22.036074+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":829,"log_dups_size":0,"ondisk_log_size":829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54394880,"num_objects":830,"num_object_clones":0,"num_object_copies":2490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":830,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":830,"num_write_kb":53120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":830}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"45'742","reported_seq":760,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:23.171877+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:23.171877+0000","last_peered":"2024-01-11T10:15:23.171877+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:23.171877+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":742,"log_dups_size":0,"ondisk_log_size":742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693248,"num_objects":743,"num_object_clones":0,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":743,"num_write_kb":47552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":743}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"45'855","reported_seq":873,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:22.888868+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:22.888868+0000","last_peered":"2024-01-11T10:15:22.888868+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:22.888868+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":54784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":856}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":205,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:58.228718+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:58.228718+0000","last_peered":"2024-01-11T10:14:58.228718+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:58.228718+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.101706+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:58.101706+0000","last_peered":"2024-01-11T10:14:58.101706+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:58.101706+0000","last_undegraded":"2024-01-11T10:14:58.101706+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"45'787","reported_seq":805,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:23.195288+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:23.195288+0000","last_peered":"2024-01-11T10:15:23.195288+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:23.195288+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":787,"log_dups_size":0,"ondisk_log_size":787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51642368,"num_objects":788,"num_object_clones":0,"num_object_copies":2364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":788,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":788,"num_write_kb":50432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":788}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.320325+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:58.320325+0000","last_peered":"2024-01-11T10:14:58.320325+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:58.320325+0000","last_undegraded":"2024-01-11T10:14:58.320325+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"45'784","reported_seq":806,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:22.146309+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:22.146309+0000","last_peered":"2024-01-11T10:15:22.146309+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:22.146309+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":784,"log_dups_size":0,"ondisk_log_size":784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":785,"num_write_kb":50240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":785}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.114342+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:58.114342+0000","last_peered":"2024-01-11T10:14:58.114342+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:58.114342+0000","last_undegraded":"2024-01-11T10:14:58.114342+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"45'764","reported_seq":786,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:22.191811+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:22.191811+0000","last_peered":"2024-01-11T10:15:22.191811+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:22.191811+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":764,"log_dups_size":0,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50135040,"num_objects":765,"num_object_clones":0,"num_object_copies":2295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":765,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":765,"num_write_kb":48960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":765}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.332897+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:58.332897+0000","last_peered":"2024-01-11T10:14:58.332897+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:58.332897+0000","last_undegraded":"2024-01-11T10:14:58.332897+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"45'850","reported_seq":872,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:21.724416+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:21.724416+0000","last_peered":"2024-01-11T10:15:21.724416+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:21.724416+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":850,"log_dups_size":0,"ondisk_log_size":850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55771136,"num_objects":851,"num_object_clones":0,"num_object_copies":2553,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":851,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":851,"num_write_kb":54464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":851}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.236679+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:58.236679+0000","last_peered":"2024-01-11T10:14:58.236679+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:58.236679+0000","last_undegraded":"2024-01-11T10:14:58.236679+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"45'743","reported_seq":763,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:23.158063+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:23.158063+0000","last_peered":"2024-01-11T10:15:23.158063+0000","last_clean":"2024-01-11T10:15:23.158063+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:23.158063+0000","last_undegraded":"2024-01-11T10:15:23.158063+0000","last_fullsized":"2024-01-11T10:15:23.158063+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48758784,"num_objects":744,"num_object_clones":0,"num_object_copies":2232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":744,"num_write_kb":47616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.247746+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:58.247746+0000","last_peered":"2024-01-11T10:14:58.247746+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:58.247746+0000","last_undegraded":"2024-01-11T10:14:58.247746+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"45'794","reported_seq":816,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:21.910116+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:21.910116+0000","last_peered":"2024-01-11T10:15:21.910116+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:21.910116+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52101120,"num_objects":795,"num_object_clones":0,"num_object_copies":2385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":795,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":795,"num_write_kb":50880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":795}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.325681+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:58.325681+0000","last_peered":"2024-01-11T10:14:58.325681+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:58.325681+0000","last_undegraded":"2024-01-11T10:14:58.325681+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"45'834","reported_seq":852,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:23.054977+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:23.054977+0000","last_peered":"2024-01-11T10:15:23.054977+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:23.054977+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54722560,"num_objects":835,"num_object_clones":0,"num_object_copies":2505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":835,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":835,"num_write_kb":53440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":835}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.108498+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:58.108498+0000","last_peered":"2024-01-11T10:14:58.108498+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:58.108498+0000","last_undegraded":"2024-01-11T10:14:58.108498+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":842530816,"num_objects":12856,"num_object_clones":0,"num_object_copies":38568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11367,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12856,"num_write_kb":822784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12840,"ondisk_log_size":12840,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706089,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153784,"kb_used_data":42288,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94218056,"statfs":{"total":96636764160,"available":96479289344,"internally_reserved":0,"allocated":43302912,"data_stored":686493590,"data_compressed":32593025,"data_compressed_allocated":42901504,"data_compressed_original":686342144,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706088,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":138376,"kb_used_data":35200,"kb_used_omap":0,"kb_used_meta":103168,"kb_avail":94233464,"statfs":{"total":96636764160,"available":96495067136,"internally_reserved":0,"allocated":36044800,"data_stored":570363798,"data_compressed":27078689,"data_compressed_allocated":35643392,"data_compressed_original":570212352,"omap_allocated":0,"internal_metadata":105644032},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706088,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":143196,"kb_used_data":31700,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94228644,"statfs":{"total":96636764160,"available":96490131456,"internally_reserved":0,"allocated":32460800,"data_stored":513412470,"data_compressed":24372520,"data_compressed_allocated":32079872,"data_compressed_original":513277952,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:27.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:28.005 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:28.022+0000 7f8f192ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:28.008 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:28.027+0000 7f8f192ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:28.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 70 16 13840 13824 12.3418 11.4375 0.0766548 0.0809475 2024-01-11T10:15:28.122 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:28.160 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:29.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 71 16 14039 14023 12.3432 12.4375 0.0784768 0.0809582 2024-01-11T10:15:30.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 72 16 14217 14201 12.3262 11.125 0.0541653 0.0810429 2024-01-11T10:15:30.161 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:30.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:30.290+0000 7fc845f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:30.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:30.295+0000 7fc845f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:30.422 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:30.449 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:30.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:30.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~187s 2024-01-11T10:15:31.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 73 16 14412 14396 12.3243 12.1875 0.0471805 0.0810974 2024-01-11T10:15:32.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 74 16 14609 14593 12.3242 12.3125 0.027823 0.0811061 2024-01-11T10:15:32.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:32.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:32.581+0000 7fab5b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:32.567 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:32.586+0000 7fab5b39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:32.689 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:32.690 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:15:32.721 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:32.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:32.813 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:32.824+0000 7f60cf4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:32.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:32.836+0000 7f60cf4f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:33.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 75 16 14782 14766 12.304 10.8125 0.117492 0.0811993 2024-01-11T10:15:34.033 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:15:34.049 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:15:34.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 76 16 14961 14945 12.2893 11.1875 0.159091 0.0812628 2024-01-11T10:15:34.168 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:34.178+0000 7f74b2e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:34.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:34.191+0000 7f74b2e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:34.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:34.860 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:34.877+0000 7f124a6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:34.863 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:34.882+0000 7f124a6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:34.975 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:34.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:34.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:35.034 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:15:35.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:15:35.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 77 16 15132 15116 12.2685 10.6875 0.143302 0.0814453 2024-01-11T10:15:35.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:35.225+0000 7f4bdb39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:35.217 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:35.235+0000 7f4bdb39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:36.038 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:15:36.055 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:15:36.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 78 16 15307 15291 12.2514 10.9375 0.149384 0.0815363 2024-01-11T10:15:36.218 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:36.230+0000 7f298d664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:36.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:36.240+0000 7f298d664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:36.297 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~193s 2024-01-11T10:15:36.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:37.042 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:15:37.060 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:15:37.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:37.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 79 16 15474 15458 12.2284 10.4375 0.0974486 0.0817054 2024-01-11T10:15:37.140 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.157+0000 7f432555e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.144 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.163+0000 7f432555e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.202 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.217+0000 7fb7d055e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.206 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.225+0000 7fb7d055e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.322 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:37.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:37.431 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:37.431 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:37.445 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-01-11T10:15:37.062727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":965542432,"num_objects":14726,"num_object_clones":0,"num_object_copies":44178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":14861,"num_write_kb":944230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14805,"ondisk_log_size":14805,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":524300,"kb_used_data":137964,"kb_used_omap":0,"kb_used_meta":386304,"kb_avail":376963060,"statfs":{"total":386547056640,"available":386010173440,"internally_reserved":0,"allocated":141275136,"data_stored":2237492170,"data_compressed":106227829,"data_compressed_allocated":139825152,"data_compressed_original":2236956672,"omap_allocated":0,"internal_metadata":395575296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61669376,"num_objects":941,"num_object_clones":0,"num_object_copies":2823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":60224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.879159"},"pg_stats":[{"pgid":"5.f","version":"45'978","reported_seq":996,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.193866+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:33.193866+0000","last_peered":"2024-01-11T10:15:33.193866+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:33.193866+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64159744,"num_objects":979,"num_object_clones":0,"num_object_copies":2937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":979,"num_write_kb":62656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"45'921","reported_seq":939,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.244018+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:33.244018+0000","last_peered":"2024-01-11T10:15:33.244018+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:33.244018+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60424192,"num_objects":922,"num_object_clones":0,"num_object_copies":2766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":922,"num_write_kb":59008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"45'981","reported_seq":999,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.193425+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:33.193425+0000","last_peered":"2024-01-11T10:15:33.193425+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:33.193425+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64356352,"num_objects":982,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":982,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"45'884","reported_seq":906,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:31.918957+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:31.918957+0000","last_peered":"2024-01-11T10:15:31.918957+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:31.918957+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57999360,"num_objects":885,"num_object_clones":0,"num_object_copies":2655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":885,"num_write_kb":56640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"45'868","reported_seq":888,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:33.069552+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:33.069552+0000","last_peered":"2024-01-11T10:15:33.069552+0000","last_clean":"2024-01-11T10:15:33.069552+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:33.069552+0000","last_undegraded":"2024-01-11T10:15:33.069552+0000","last_fullsized":"2024-01-11T10:15:33.069552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56950784,"num_objects":869,"num_object_clones":0,"num_object_copies":2607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":55616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"45'919","reported_seq":941,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.057505+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:32.057505+0000","last_peered":"2024-01-11T10:15:32.057505+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:32.057505+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":58880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"45'946","reported_seq":964,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.069328+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:32.069328+0000","last_peered":"2024-01-11T10:15:32.069328+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:32.069328+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62062592,"num_objects":947,"num_object_clones":0,"num_object_copies":2841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":60608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":947}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"45'866","reported_seq":884,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.112018+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:33.112018+0000","last_peered":"2024-01-11T10:15:33.112018+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:33.112018+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56819712,"num_objects":867,"num_object_clones":0,"num_object_copies":2601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":867,"num_write_kb":55488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"45'987","reported_seq":1005,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.197670+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:33.197670+0000","last_peered":"2024-01-11T10:15:33.197670+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:33.197670+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":63232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":205,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:58.228718+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:58.228718+0000","last_peered":"2024-01-11T10:14:58.228718+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:58.228718+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.101706+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:58.101706+0000","last_peered":"2024-01-11T10:14:58.101706+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:58.101706+0000","last_undegraded":"2024-01-11T10:14:58.101706+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"45'903","reported_seq":921,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.167783+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:33.167783+0000","last_peered":"2024-01-11T10:15:33.167783+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:33.167783+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":903,"log_dups_size":0,"ondisk_log_size":903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":57856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":904}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.320325+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:58.320325+0000","last_peered":"2024-01-11T10:14:58.320325+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:58.320325+0000","last_undegraded":"2024-01-11T10:14:58.320325+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"45'901","reported_seq":923,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.164172+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:32.164172+0000","last_peered":"2024-01-11T10:15:32.164172+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:32.164172+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59113472,"num_objects":902,"num_object_clones":0,"num_object_copies":2706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":902,"num_write_kb":57728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.114342+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:58.114342+0000","last_peered":"2024-01-11T10:14:58.114342+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:58.114342+0000","last_undegraded":"2024-01-11T10:14:58.114342+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"45'874","reported_seq":896,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.105477+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:32.105477+0000","last_peered":"2024-01-11T10:15:32.105477+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:32.105477+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":56000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.332897+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:58.332897+0000","last_peered":"2024-01-11T10:14:58.332897+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:58.332897+0000","last_undegraded":"2024-01-11T10:14:58.332897+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"45'974","reported_seq":996,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.091568+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:32.091568+0000","last_peered":"2024-01-11T10:15:32.091568+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:32.091568+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63897600,"num_objects":975,"num_object_clones":0,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":975,"num_write_kb":62400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.236679+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:58.236679+0000","last_peered":"2024-01-11T10:14:58.236679+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:58.236679+0000","last_undegraded":"2024-01-11T10:14:58.236679+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"45'855","reported_seq":875,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:32.987865+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:32.987865+0000","last_peered":"2024-01-11T10:15:32.987865+0000","last_clean":"2024-01-11T10:15:32.987865+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:32.987865+0000","last_undegraded":"2024-01-11T10:15:32.987865+0000","last_fullsized":"2024-01-11T10:15:32.987865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":54784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.247746+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:58.247746+0000","last_peered":"2024-01-11T10:14:58.247746+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:58.247746+0000","last_undegraded":"2024-01-11T10:14:58.247746+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"45'904","reported_seq":926,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.077669+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:32.077669+0000","last_peered":"2024-01-11T10:15:32.077669+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:32.077669+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":904,"log_dups_size":0,"ondisk_log_size":904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":905,"num_write_kb":57920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.325681+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:58.325681+0000","last_peered":"2024-01-11T10:14:58.325681+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:58.325681+0000","last_undegraded":"2024-01-11T10:14:58.325681+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"45'947","reported_seq":965,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.211922+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:33.211922+0000","last_peered":"2024-01-11T10:15:33.211922+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:33.211922+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62128128,"num_objects":948,"num_object_clones":0,"num_object_copies":2844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":948,"num_write_kb":60672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":948}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.108498+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:58.108498+0000","last_peered":"2024-01-11T10:14:58.108498+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:58.108498+0000","last_undegraded":"2024-01-11T10:14:58.108498+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":964952064,"num_objects":14724,"num_object_clones":0,"num_object_copies":44172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":942336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14708,"ondisk_log_size":14708,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":147460,"kb_used_data":48444,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94224380,"statfs":{"total":96636764160,"available":96485765120,"internally_reserved":0,"allocated":49606656,"data_stored":787353494,"data_compressed":37382393,"data_compressed_allocated":49205248,"data_compressed_original":787202048,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165644,"kb_used_data":40260,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94206196,"statfs":{"total":96636764160,"available":96467144704,"internally_reserved":0,"allocated":41226240,"data_stored":653266838,"data_compressed":31015369,"data_compressed_allocated":40824832,"data_compressed_original":653115392,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135420,"kb_used_data":36404,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94236420,"statfs":{"total":96636764160,"available":96498094080,"internally_reserved":0,"allocated":37277696,"data_stored":590482806,"data_compressed":28032232,"data_compressed_allocated":36896768,"data_compressed_original":590348288,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:37.447 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:37.594 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.609+0000 7f7c4699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.617+0000 7f7c4699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.808 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:37.809 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:37.826 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-01-11T10:15:37.062727+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":965542432,"num_objects":14726,"num_object_clones":0,"num_object_copies":44178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13001,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":14861,"num_write_kb":944230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14805,"ondisk_log_size":14805,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":524300,"kb_used_data":137964,"kb_used_omap":0,"kb_used_meta":386304,"kb_avail":376963060,"statfs":{"total":386547056640,"available":386010173440,"internally_reserved":0,"allocated":141275136,"data_stored":2237492170,"data_compressed":106227829,"data_compressed_allocated":139825152,"data_compressed_original":2236956672,"omap_allocated":0,"internal_metadata":395575296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":61669376,"num_objects":941,"num_object_clones":0,"num_object_copies":2823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":941,"num_write_kb":60224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.879159"},"pg_stats":[{"pgid":"5.f","version":"45'978","reported_seq":996,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.193866+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:33.193866+0000","last_peered":"2024-01-11T10:15:33.193866+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:33.193866+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64159744,"num_objects":979,"num_object_clones":0,"num_object_copies":2937,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":979,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":979,"num_write_kb":62656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":979}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"45'921","reported_seq":939,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.244018+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:33.244018+0000","last_peered":"2024-01-11T10:15:33.244018+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:33.244018+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60424192,"num_objects":922,"num_object_clones":0,"num_object_copies":2766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":922,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":922,"num_write_kb":59008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":922}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"45'981","reported_seq":999,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.193425+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:33.193425+0000","last_peered":"2024-01-11T10:15:33.193425+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:33.193425+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":981,"log_dups_size":0,"ondisk_log_size":981,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64356352,"num_objects":982,"num_object_clones":0,"num_object_copies":2946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":982,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":982,"num_write_kb":62848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":982}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"45'884","reported_seq":906,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:31.918957+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:31.918957+0000","last_peered":"2024-01-11T10:15:31.918957+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:31.918957+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":884,"log_dups_size":0,"ondisk_log_size":884,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57999360,"num_objects":885,"num_object_clones":0,"num_object_copies":2655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":885,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":885,"num_write_kb":56640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":885}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"45'868","reported_seq":888,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:33.069552+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:33.069552+0000","last_peered":"2024-01-11T10:15:33.069552+0000","last_clean":"2024-01-11T10:15:33.069552+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:33.069552+0000","last_undegraded":"2024-01-11T10:15:33.069552+0000","last_fullsized":"2024-01-11T10:15:33.069552+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":868,"log_dups_size":0,"ondisk_log_size":868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56950784,"num_objects":869,"num_object_clones":0,"num_object_copies":2607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":869,"num_write_kb":55616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"45'919","reported_seq":941,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.057505+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:32.057505+0000","last_peered":"2024-01-11T10:15:32.057505+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:32.057505+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":920,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":58880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":920}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"45'946","reported_seq":964,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.069328+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:32.069328+0000","last_peered":"2024-01-11T10:15:32.069328+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:32.069328+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62062592,"num_objects":947,"num_object_clones":0,"num_object_copies":2841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":947,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":947,"num_write_kb":60608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":947}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"45'866","reported_seq":884,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.112018+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:33.112018+0000","last_peered":"2024-01-11T10:15:33.112018+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:33.112018+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":866,"log_dups_size":0,"ondisk_log_size":866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56819712,"num_objects":867,"num_object_clones":0,"num_object_copies":2601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":867,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":867,"num_write_kb":55488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":867}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"45'987","reported_seq":1005,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.197670+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:33.197670+0000","last_peered":"2024-01-11T10:15:33.197670+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:33.197670+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":987,"log_dups_size":0,"ondisk_log_size":987,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64749568,"num_objects":988,"num_object_clones":0,"num_object_copies":2964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":988,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":988,"num_write_kb":63232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":988}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":205,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:14:58.228718+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:14:58.228718+0000","last_peered":"2024-01-11T10:14:58.228718+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:14:58.228718+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.101706+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:14:58.101706+0000","last_peered":"2024-01-11T10:14:58.101706+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:14:58.101706+0000","last_undegraded":"2024-01-11T10:14:58.101706+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"45'903","reported_seq":921,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.167783+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:33.167783+0000","last_peered":"2024-01-11T10:15:33.167783+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:33.167783+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":903,"log_dups_size":0,"ondisk_log_size":903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59244544,"num_objects":904,"num_object_clones":0,"num_object_copies":2712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":904,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":904,"num_write_kb":57856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":904}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.320325+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:14:58.320325+0000","last_peered":"2024-01-11T10:14:58.320325+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:14:58.320325+0000","last_undegraded":"2024-01-11T10:14:58.320325+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"45'901","reported_seq":923,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.164172+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:32.164172+0000","last_peered":"2024-01-11T10:15:32.164172+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:32.164172+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":901,"log_dups_size":0,"ondisk_log_size":901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59113472,"num_objects":902,"num_object_clones":0,"num_object_copies":2706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":902,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":902,"num_write_kb":57728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":902}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.114342+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:14:58.114342+0000","last_peered":"2024-01-11T10:14:58.114342+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:14:58.114342+0000","last_undegraded":"2024-01-11T10:14:58.114342+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"45'874","reported_seq":896,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.105477+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:32.105477+0000","last_peered":"2024-01-11T10:15:32.105477+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:32.105477+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":874,"log_dups_size":0,"ondisk_log_size":874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57344000,"num_objects":875,"num_object_clones":0,"num_object_copies":2625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":875,"num_write_kb":56000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":875}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":60,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.332897+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:14:58.332897+0000","last_peered":"2024-01-11T10:14:58.332897+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:14:58.332897+0000","last_undegraded":"2024-01-11T10:14:58.332897+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"45'974","reported_seq":996,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.091568+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:32.091568+0000","last_peered":"2024-01-11T10:15:32.091568+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:32.091568+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63897600,"num_objects":975,"num_object_clones":0,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":975,"num_write_kb":62400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":975}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":37,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.236679+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:14:58.236679+0000","last_peered":"2024-01-11T10:14:58.236679+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:14:58.236679+0000","last_undegraded":"2024-01-11T10:14:58.236679+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"45'855","reported_seq":875,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-01-11T10:15:32.987865+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:32.987865+0000","last_peered":"2024-01-11T10:15:32.987865+0000","last_clean":"2024-01-11T10:15:32.987865+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:32.987865+0000","last_undegraded":"2024-01-11T10:15:32.987865+0000","last_fullsized":"2024-01-11T10:15:32.987865+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56098816,"num_objects":856,"num_object_clones":0,"num_object_copies":2568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":856,"num_write_kb":54784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":74,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.247746+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:14:58.247746+0000","last_peered":"2024-01-11T10:14:58.247746+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:14:58.247746+0000","last_undegraded":"2024-01-11T10:14:58.247746+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"45'904","reported_seq":926,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:32.077669+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:32.077669+0000","last_peered":"2024-01-11T10:15:32.077669+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:32.077669+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":904,"log_dups_size":0,"ondisk_log_size":904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":905,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":905,"num_write_kb":57920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":905}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":29,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.325681+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:14:58.325681+0000","last_peered":"2024-01-11T10:14:58.325681+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:14:58.325681+0000","last_undegraded":"2024-01-11T10:14:58.325681+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"45'947","reported_seq":965,"reported_epoch":45,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:33.211922+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:33.211922+0000","last_peered":"2024-01-11T10:15:33.211922+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:33.211922+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":947,"log_dups_size":0,"ondisk_log_size":947,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62128128,"num_objects":948,"num_object_clones":0,"num_object_copies":2844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":948,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":948,"num_write_kb":60672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":948}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":45,"state":"active+undersized","last_fresh":"2024-01-11T10:14:58.108498+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:14:58.108498+0000","last_peered":"2024-01-11T10:14:58.108498+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:14:58.108498+0000","last_undegraded":"2024-01-11T10:14:58.108498+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":964952064,"num_objects":14724,"num_object_clones":0,"num_object_copies":44172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14724,"num_write_kb":942336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14708,"ondisk_log_size":14708,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706099,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":147460,"kb_used_data":48444,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94224380,"statfs":{"total":96636764160,"available":96485765120,"internally_reserved":0,"allocated":49606656,"data_stored":787353494,"data_compressed":37382393,"data_compressed_allocated":49205248,"data_compressed_original":787202048,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165644,"kb_used_data":40260,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94206196,"statfs":{"total":96636764160,"available":96467144704,"internally_reserved":0,"allocated":41226240,"data_stored":653266838,"data_compressed":31015369,"data_compressed_allocated":40824832,"data_compressed_original":653115392,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706098,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135420,"kb_used_data":36404,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94236420,"statfs":{"total":96636764160,"available":96498094080,"internally_reserved":0,"allocated":37277696,"data_stored":590482806,"data_compressed":28032232,"data_compressed_allocated":36896768,"data_compressed_original":590348288,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:37.827 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:15:37.827 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:37.827 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:15:37.827 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:15:37.949 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.963+0000 7f807bb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:37.953 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:37.972+0000 7f807bb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:38.085 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:38.088 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:38.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:15:38.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:15:38.133669+0000 min lat: 0.00835493 max lat: 0.30334 avg lat: 0.0818952 2024-01-11T10:15:38.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:15:38.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 80 16 15638 15622 12.2037 10.25 0.0170721 0.0818952 2024-01-11T10:15:38.208 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:38.222+0000 7fd7d482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.212 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:38.231+0000 7fd7d482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:38.230 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:38.231 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:38.234 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:38.234 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:15:38.394 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:38.408+0000 7f37b075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.397 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:38.416+0000 7f37b075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.402 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:38.406 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:38.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:15:38.535 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:38.546+0000 7fb842d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.536 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:38.555+0000 7fb842d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:38.550 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:38.551 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:38.553 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:38.554 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:39.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 81 16 15806 15790 12.1826 10.5 0.136422 0.0820034 2024-01-11T10:15:39.346 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:39.470 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:39.487+0000 7fc61439e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:39.474 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:39.493+0000 7fc61439e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:39.594 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:39.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:40.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 82 16 15982 15966 12.1682 11 0.0205294 0.0821084 2024-01-11T10:15:41.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 83 16 16183 16167 12.1729 12.5625 0.020887 0.0821068 2024-01-11T10:15:41.618 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:41.740 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:41.757+0000 7f440e464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:41.743 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:41.762+0000 7f440e464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:41.801 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~198s 2024-01-11T10:15:41.878 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:41.899 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:42.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 84 16 16370 16354 12.1671 11.6875 0.143021 0.0821325 2024-01-11T10:15:43.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 85 16 16559 16543 12.163 11.8125 0.0170115 0.082181 2024-01-11T10:15:43.555 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:43.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:43.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:43.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:43.690+0000 7f73d679e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:43.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:43.697+0000 7f73d679e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:43.887 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:43.887 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:43.899 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:43.900 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-01-11T10:15:42.186296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1021837856,"num_objects":15585,"num_object_clones":0,"num_object_copies":46755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":15720,"num_write_kb":999206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15664,"ondisk_log_size":15664,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":535776,"kb_used_data":145280,"kb_used_omap":0,"kb_used_meta":390464,"kb_avail":376951584,"statfs":{"total":386547056640,"available":385998422016,"internally_reserved":0,"allocated":148766720,"data_stored":2356079244,"data_compressed":111857437,"data_compressed_allocated":147234816,"data_compressed_original":2355511296,"omap_allocated":0,"internal_metadata":399835136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":56295424,"num_objects":859,"num_object_clones":0,"num_object_copies":2577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":755,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":859,"num_write_kb":54976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.132532"},"pg_stats":[{"pgid":"5.f","version":"49'1037","reported_seq":1061,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:38.117592+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:38.117592+0000","last_peered":"2024-01-11T10:15:38.117592+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:38.117592+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1037,"log_dups_size":0,"ondisk_log_size":1037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68026368,"num_objects":1038,"num_object_clones":0,"num_object_copies":3114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1038,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1038,"num_write_kb":66432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1038}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"49'977","reported_seq":1001,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:38.244227+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:38.244227+0000","last_peered":"2024-01-11T10:15:38.244227+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:38.244227+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":977,"log_dups_size":0,"ondisk_log_size":977,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64094208,"num_objects":978,"num_object_clones":0,"num_object_copies":2934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":978,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":978,"num_write_kb":62592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":978}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"49'1023","reported_seq":1047,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:38.172039+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:38.172039+0000","last_peered":"2024-01-11T10:15:38.172039+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:38.172039+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":1024,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1024,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1024}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"48'933","reported_seq":961,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.061113+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:37.061113+0000","last_peered":"2024-01-11T10:15:37.061113+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:37.061113+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":933,"log_dups_size":0,"ondisk_log_size":933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61210624,"num_objects":934,"num_object_clones":0,"num_object_copies":2802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":934,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":934,"num_write_kb":59776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":934}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"49'922","reported_seq":948,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-01-11T10:15:37.966459+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:37.966459+0000","last_peered":"2024-01-11T10:15:37.966459+0000","last_clean":"2024-01-11T10:15:37.966459+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:37.966459+0000","last_undegraded":"2024-01-11T10:15:37.966459+0000","last_fullsized":"2024-01-11T10:15:37.966459+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":922,"log_dups_size":0,"ondisk_log_size":922,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60489728,"num_objects":923,"num_object_clones":0,"num_object_copies":2769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":923,"num_write_kb":59072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"48'976","reported_seq":1004,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.040827+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:37.040827+0000","last_peered":"2024-01-11T10:15:37.040827+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:37.040827+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":976,"log_dups_size":0,"ondisk_log_size":976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64028672,"num_objects":977,"num_object_clones":0,"num_object_copies":2931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":977,"num_write_kb":62528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":977}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"49'993","reported_seq":1017,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.199335+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:37.199335+0000","last_peered":"2024-01-11T10:15:37.199335+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:37.199335+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":993,"log_dups_size":0,"ondisk_log_size":993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65142784,"num_objects":994,"num_object_clones":0,"num_object_copies":2982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":994,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":994,"num_write_kb":63616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":994}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"49'928","reported_seq":952,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:38.072019+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:38.072019+0000","last_peered":"2024-01-11T10:15:38.072019+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:38.072019+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":928,"log_dups_size":0,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60882944,"num_objects":929,"num_object_clones":0,"num_object_copies":2787,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":929,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":929,"num_write_kb":59456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":929}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"49'1043","reported_seq":1067,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:38.206488+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:38.206488+0000","last_peered":"2024-01-11T10:15:38.206488+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:38.206488+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1043,"log_dups_size":0,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68419584,"num_objects":1044,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1044,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":66816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1044}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":211,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:36.079407+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:15:36.079407+0000","last_peered":"2024-01-11T10:15:36.079407+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:15:36.079407+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":68,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.146081+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:15:37.146081+0000","last_peered":"2024-01-11T10:15:37.146081+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:15:37.146081+0000","last_undegraded":"2024-01-11T10:15:37.146081+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"49'961","reported_seq":985,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.923519+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:37.923519+0000","last_peered":"2024-01-11T10:15:37.923519+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:37.923519+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63045632,"num_objects":962,"num_object_clones":0,"num_object_copies":2886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":962,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":962,"num_write_kb":61568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":962}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.361984+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:15:37.361984+0000","last_peered":"2024-01-11T10:15:37.361984+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:15:37.361984+0000","last_undegraded":"2024-01-11T10:15:37.361984+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"48'945","reported_seq":973,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.022526+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:37.022526+0000","last_peered":"2024-01-11T10:15:37.022526+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:37.022526+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":945,"log_dups_size":0,"ondisk_log_size":945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61997056,"num_objects":946,"num_object_clones":0,"num_object_copies":2838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":946,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":946,"num_write_kb":60544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":946}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.129366+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:15:37.129366+0000","last_peered":"2024-01-11T10:15:37.129366+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:15:37.129366+0000","last_undegraded":"2024-01-11T10:15:37.129366+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"48'939","reported_seq":967,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.138168+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:37.138168+0000","last_peered":"2024-01-11T10:15:37.138168+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:37.138168+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":939,"log_dups_size":0,"ondisk_log_size":939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61603840,"num_objects":940,"num_object_clones":0,"num_object_copies":2820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":940,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":940,"num_write_kb":60160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":940}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":66,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.339783+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:15:37.339783+0000","last_peered":"2024-01-11T10:15:37.339783+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:15:37.339783+0000","last_undegraded":"2024-01-11T10:15:37.339783+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"48'1023","reported_seq":1051,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.046685+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:37.046685+0000","last_peered":"2024-01-11T10:15:37.046685+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:37.046685+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":1024,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1024,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1024,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1024}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":43,"reported_epoch":48,"state":"active+undersized","last_fresh":"2024-01-11T10:15:36.080396+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:15:36.080396+0000","last_peered":"2024-01-11T10:15:36.080396+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:15:36.080396+0000","last_undegraded":"2024-01-11T10:15:36.080396+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"49'905","reported_seq":931,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-01-11T10:15:37.956849+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:37.956849+0000","last_peered":"2024-01-11T10:15:37.956849+0000","last_clean":"2024-01-11T10:15:37.956849+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:37.956849+0000","last_undegraded":"2024-01-11T10:15:37.956849+0000","last_fullsized":"2024-01-11T10:15:37.956849+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":905,"log_dups_size":0,"ondisk_log_size":905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59375616,"num_objects":906,"num_object_clones":0,"num_object_copies":2718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":906,"num_write_kb":57984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":80,"reported_epoch":48,"state":"active+undersized","last_fresh":"2024-01-11T10:15:36.072467+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:15:36.072467+0000","last_peered":"2024-01-11T10:15:36.072467+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:15:36.072467+0000","last_undegraded":"2024-01-11T10:15:36.072467+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"48'963","reported_seq":991,"reported_epoch":48,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:36.981877+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:36.981877+0000","last_peered":"2024-01-11T10:15:36.981877+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:36.981877+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":963,"log_dups_size":0,"ondisk_log_size":963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63176704,"num_objects":964,"num_object_clones":0,"num_object_copies":2892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":964,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":964,"num_write_kb":61696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":964}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":35,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.371722+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:15:37.371722+0000","last_peered":"2024-01-11T10:15:37.371722+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:15:37.371722+0000","last_undegraded":"2024-01-11T10:15:37.371722+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"49'999","reported_seq":1023,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:38.131938+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:38.131938+0000","last_peered":"2024-01-11T10:15:38.131938+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:38.131938+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536000,"num_objects":1000,"num_object_clones":0,"num_object_copies":3000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1000,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1000,"num_write_kb":64000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1000}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.121429+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:15:37.121429+0000","last_peered":"2024-01-11T10:15:37.121429+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:15:37.121429+0000","last_undegraded":"2024-01-11T10:15:37.121429+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1021247488,"num_objects":15583,"num_object_clones":0,"num_object_copies":46749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13754,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15583,"num_write_kb":997312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15567,"ondisk_log_size":15567,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706104,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":154412,"kb_used_data":51236,"kb_used_omap":0,"kb_used_meta":103168,"kb_avail":94217428,"statfs":{"total":96636764160,"available":96478646272,"internally_reserved":0,"allocated":52465664,"data_stored":832586314,"data_compressed":39529673,"data_compressed_allocated":52031488,"data_compressed_original":832421888,"omap_allocated":0,"internal_metadata":105644032},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706103,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":167960,"kb_used_data":42576,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94203880,"statfs":{"total":96636764160,"available":96464773120,"internally_reserved":0,"allocated":43597824,"data_stored":690828701,"data_compressed":32798545,"data_compressed_allocated":43171840,"data_compressed_original":690667520,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706103,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":137628,"kb_used_data":38612,"kb_used_omap":0,"kb_used_meta":99008,"kb_avail":94234212,"statfs":{"total":96636764160,"available":96495833088,"internally_reserved":0,"allocated":39538688,"data_stored":626275197,"data_compressed":29731384,"data_compressed_allocated":39133184,"data_compressed_original":626130944,"omap_allocated":0,"internal_metadata":101384192},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:43.902 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 5.e 5.c 1.0 5.6 5.5 5.3 5.1 5.0 2024-01-11T10:15:44.061 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:44.077+0000 7f8f8359e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:44.065 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:44.083+0000 7f8f8359e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:44.086 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:44.092+0000 7fae4039e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:44.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:44.108+0000 7fae4039e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:44.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 86 16 16758 16742 12.1661 12.4375 0.163247 0.0821663 2024-01-11T10:15:44.251 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:44.274 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:44.274 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:44.349 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.0,5.e] on osd.0 to force-backfill; instructing pg(s) [1.0,5.1,5.3,5.5,5.6,5.c] on osd.3 to force-backfill; 2024-01-11T10:15:44.349 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:15:45.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 87 16 16939 16923 12.1563 11.3125 0.108596 0.0822225 2024-01-11T10:15:46.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 88 16 17111 17095 12.1403 10.75 0.0583353 0.0823121 2024-01-11T10:15:46.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:46.387 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:46.404+0000 7efe0895e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:46.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:46.410+0000 7efe0895e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:46.549 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:46.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:47.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 89 16 17282 17266 12.124 10.6875 0.121454 0.0824512 2024-01-11T10:15:47.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~204s 2024-01-11T10:15:48.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 90 11 17473 17462 12.1254 12.25 0.14268 0.0824091 2024-01-11T10:15:48.567 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:48.675 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:48.692+0000 7f4cb3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:48.678 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:48.696+0000 7f4cb3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:48.710 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:48.728 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:48.728 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:49.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Total time run: 90.0216 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Total writes made: 17473 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Write size: 65536 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object size: 65536 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Bandwidth (MB/sec): 12.1311 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev Bandwidth: 0.938739 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max bandwidth (MB/sec): 14.25 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min bandwidth (MB/sec): 10.25 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Average IOPS: 194 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev IOPS: 15.0198 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max IOPS: 228 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min IOPS: 164 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Average Latency(s): 0.0824172 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Stddev Latency(s): 0.0455173 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Max latency(s): 0.30334 2024-01-11T10:15:49.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Min latency(s): 0.00835493 2024-01-11T10:15:49.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Cleaning up (deleting benchmark objects) 2024-01-11T10:15:49.368 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:49.368 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:15:49.368 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:15:49.368 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:15:49.484 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:49.497+0000 7fc23fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:49.487 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:49.505+0000 7fc23fe6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:49.585 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:49.586 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:49.589 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:49.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:15:49.709 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:49.722+0000 7f942d39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:49.730+0000 7f942d39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:49.742 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:49.742 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:49.742 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:49.742 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:49.742 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:49.743 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:49.745 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:49.746 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:15:49.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:49.874+0000 7f3c7451e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:49.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:49.883+0000 7f3c7451e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:49.870 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:49.876 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:49.876 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:15:50.017 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:50.030+0000 7fd6fc2eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:50.021 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:50.039+0000 7fd6fc2eb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:15:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:15:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:15:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:15:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:15:50.041 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:15:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:15:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:15:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:15:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:15:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:15:50.042 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:15:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:15:50.045 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:15:50.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:50.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:50.860+0000 7fbcad308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:50.848 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:50.866+0000 7fbcad308640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:50.972 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:50.996 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:52.809 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~209s 2024-01-11T10:15:52.997 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:53.117 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:53.134+0000 7f236ab5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:53.121 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:53.138+0000 7f236ab5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:53.226 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:53.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:53.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:55.046 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:15:55.047 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:15:55.165 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:55.176+0000 7fbde671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:55.169 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:55.187+0000 7fbde671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:55.249 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:55.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:55.382+0000 7f903239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:55.369 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:55.386+0000 7f903239e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:55.488 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:55.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:56.225 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:15:56.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:15:56.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:56.375+0000 7f1d9c424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:56.366 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:56.384+0000 7f1d9c424640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:57.229 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:15:57.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:15:57.361 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:57.372+0000 7faad3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:57.365 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:57.383+0000 7faad3c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:57.510 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:15:57.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:57.638+0000 7fea1cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:57.625 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:57.643+0000 7fea1cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:57.747 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:15:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:15:57.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:15:58.233 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:15:58.252 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:15:58.313 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~215s 2024-01-11T10:15:58.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:58.397+0000 7fbba2f55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:58.391 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:58.409+0000 7fbba2f55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.236 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:15:59.254 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:15:59.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:59.389 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:59.403+0000 7f218786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.392 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:59.410+0000 7f218786d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.596 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:59.597 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:15:59.609 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-01-11T10:15:59.258509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1026359864,"num_objects":15655,"num_object_clones":0,"num_object_copies":46965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":19432,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19381,"ondisk_log_size":19381,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":558852,"kb_used_data":147552,"kb_used_omap":0,"kb_used_meta":411264,"kb_avail":376928508,"statfs":{"total":386547056640,"available":385974792192,"internally_reserved":0,"allocated":151093248,"data_stored":2392789187,"data_compressed":113602511,"data_compressed_allocated":149528576,"data_compressed_original":2392211456,"omap_allocated":0,"internal_metadata":421134336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66191360,"num_objects":-1010,"num_object_clones":0,"num_object_copies":-3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.068565"},"pg_stats":[{"pgid":"5.f","version":"49'1295","reported_seq":1321,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.272374+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:58.272374+0000","last_peered":"2024-01-11T10:15:58.272374+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:58.272374+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66715648,"num_objects":1018,"num_object_clones":0,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1296,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1018}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"49'1211","reported_seq":1235,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.820252+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:57.820252+0000","last_peered":"2024-01-11T10:15:57.820252+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:57.820252+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1211,"log_dups_size":0,"ondisk_log_size":1211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011712,"num_objects":992,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1212,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":992}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"49'1266","reported_seq":1292,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.269488+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:58.269488+0000","last_peered":"2024-01-11T10:15:58.269488+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:58.269488+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1266,"log_dups_size":0,"ondisk_log_size":1266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67436568,"num_objects":1030,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1266,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1030}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"49'1172","reported_seq":1202,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.054238+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:57.054238+0000","last_peered":"2024-01-11T10:15:57.054238+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:57.054238+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1172,"log_dups_size":0,"ondisk_log_size":1172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586880,"num_objects":955,"num_object_clones":0,"num_object_copies":2865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1173,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":955}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"49'1163","reported_seq":1190,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-01-11T10:15:58.278445+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:58.278445+0000","last_peered":"2024-01-11T10:15:58.278445+0000","last_clean":"2024-01-11T10:15:58.278445+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:58.278445+0000","last_undegraded":"2024-01-11T10:15:58.278445+0000","last_fullsized":"2024-01-11T10:15:58.278445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":2751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1163,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"49'1212","reported_seq":1242,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.154961+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:57.154961+0000","last_peered":"2024-01-11T10:15:57.154961+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:57.154961+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1213,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"50'1236","reported_seq":1262,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.087943+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:57.087943+0000","last_peered":"2024-01-11T10:15:57.087943+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:57.087943+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65732608,"num_objects":1003,"num_object_clones":0,"num_object_copies":3009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1237,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"49'1158","reported_seq":1182,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.134998+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:58.134998+0000","last_peered":"2024-01-11T10:15:58.134998+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:58.134998+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61669376,"num_objects":941,"num_object_clones":0,"num_object_copies":2823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"49'1261","reported_seq":1285,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.149066+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:58.149066+0000","last_peered":"2024-01-11T10:15:58.149066+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:58.149066+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67239936,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1262,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1026}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":213,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.251981+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:15:37.251981+0000","last_peered":"2024-01-11T10:15:37.251981+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:15:37.251981+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":70,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:56.289524+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:15:56.289524+0000","last_peered":"2024-01-11T10:15:56.289524+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:15:56.289524+0000","last_undegraded":"2024-01-11T10:15:56.289524+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"49'1185","reported_seq":1210,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.279117+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:58.279117+0000","last_peered":"2024-01-11T10:15:58.279117+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:58.279117+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1185,"log_dups_size":0,"ondisk_log_size":1185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62193664,"num_objects":949,"num_object_clones":0,"num_object_copies":2847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1185,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":949}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.361984+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:15:37.361984+0000","last_peered":"2024-01-11T10:15:37.361984+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:15:37.361984+0000","last_undegraded":"2024-01-11T10:15:37.361984+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"49'1178","reported_seq":1208,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.032070+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:57.032070+0000","last_peered":"2024-01-11T10:15:57.032070+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:57.032070+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1178,"log_dups_size":0,"ondisk_log_size":1178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63766528,"num_objects":973,"num_object_clones":0,"num_object_copies":2919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1179,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":973}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:56.276591+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:15:56.276591+0000","last_peered":"2024-01-11T10:15:56.276591+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:15:56.276591+0000","last_undegraded":"2024-01-11T10:15:56.276591+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"49'1146","reported_seq":1176,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:56.968830+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:56.968830+0000","last_peered":"2024-01-11T10:15:56.968830+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:56.968830+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1146,"log_dups_size":0,"ondisk_log_size":1146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63111168,"num_objects":963,"num_object_clones":0,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:58.268601+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:15:58.268601+0000","last_peered":"2024-01-11T10:15:58.268601+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:15:58.268601+0000","last_undegraded":"2024-01-11T10:15:58.268601+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"49'1251","reported_seq":1281,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:56.975683+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:56.975683+0000","last_peered":"2024-01-11T10:15:56.975683+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:56.975683+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67502080,"num_objects":1030,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.243016+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:15:37.243016+0000","last_peered":"2024-01-11T10:15:37.243016+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:15:37.243016+0000","last_undegraded":"2024-01-11T10:15:37.243016+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"49'1135","reported_seq":1162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-01-11T10:15:58.272912+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:58.272912+0000","last_peered":"2024-01-11T10:15:58.272912+0000","last_clean":"2024-01-11T10:15:58.272912+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:58.272912+0000","last_undegraded":"2024-01-11T10:15:58.272912+0000","last_fullsized":"2024-01-11T10:15:58.272912+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1135,"log_dups_size":0,"ondisk_log_size":1135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1135,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":82,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.252838+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:15:37.252838+0000","last_peered":"2024-01-11T10:15:37.252838+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:15:37.252838+0000","last_undegraded":"2024-01-11T10:15:37.252838+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"49'1201","reported_seq":1231,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.087386+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:57.087386+0000","last_peered":"2024-01-11T10:15:57.087386+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:57.087386+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1201,"log_dups_size":0,"ondisk_log_size":1201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1202,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:58.275316+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:15:58.275316+0000","last_peered":"2024-01-11T10:15:58.275316+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:15:58.275316+0000","last_undegraded":"2024-01-11T10:15:58.275316+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"49'1214","reported_seq":1239,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.277406+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:58.277406+0000","last_peered":"2024-01-11T10:15:58.277406+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:58.277406+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011712,"num_objects":992,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":992}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":70,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:56.299105+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:15:56.299105+0000","last_peered":"2024-01-11T10:15:56.299105+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:15:56.299105+0000","last_undegraded":"2024-01-11T10:15:56.299105+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1025769496,"num_objects":15653,"num_object_clones":0,"num_object_copies":46959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19295,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19284,"ondisk_log_size":19284,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706124,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166864,"kb_used_data":51208,"kb_used_omap":0,"kb_used_meta":115648,"kb_avail":94204976,"statfs":{"total":96636764160,"available":96465895424,"internally_reserved":0,"allocated":52436992,"data_stored":832062050,"data_compressed":39505687,"data_compressed_allocated":51998720,"data_compressed_original":831897600,"omap_allocated":0,"internal_metadata":118423552},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165112,"kb_used_data":43888,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94206728,"statfs":{"total":96636764160,"available":96467689472,"internally_reserved":0,"allocated":44941312,"data_stored":712131170,"data_compressed":33810726,"data_compressed_allocated":44503040,"data_compressed_original":711966720,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151100,"kb_used_data":39600,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94220740,"statfs":{"total":96636764160,"available":96482037760,"internally_reserved":0,"allocated":40550400,"data_stored":642206935,"data_compressed":30488263,"data_compressed_allocated":40128512,"data_compressed_original":642056192,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:15:59.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:15:59.728 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:59.742+0000 7f4db999e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.732 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:59.750+0000 7f4db999e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.764 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:15:59.880 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:59.896+0000 7f00cb6f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.885 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:15:59.903+0000 7f00cb6f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:15:59.992 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:15:59.993 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:16:00.007 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-01-11T10:15:59.258509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1026359864,"num_objects":15655,"num_object_clones":0,"num_object_copies":46965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13833,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":19432,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19381,"ondisk_log_size":19381,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":558852,"kb_used_data":147552,"kb_used_omap":0,"kb_used_meta":411264,"kb_avail":376928508,"statfs":{"total":386547056640,"available":385974792192,"internally_reserved":0,"allocated":151093248,"data_stored":2392789187,"data_compressed":113602511,"data_compressed_allocated":149528576,"data_compressed_original":2392211456,"omap_allocated":0,"internal_metadata":421134336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-66191360,"num_objects":-1010,"num_object_clones":0,"num_object_copies":-3030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-880,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1010,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.068565"},"pg_stats":[{"pgid":"5.f","version":"49'1295","reported_seq":1321,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.272374+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:15:58.272374+0000","last_peered":"2024-01-11T10:15:58.272374+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:15:58.272374+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66715648,"num_objects":1018,"num_object_clones":0,"num_object_copies":3054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1296,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1018}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"49'1211","reported_seq":1235,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.820252+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:15:57.820252+0000","last_peered":"2024-01-11T10:15:57.820252+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:15:57.820252+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1211,"log_dups_size":0,"ondisk_log_size":1211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011712,"num_objects":992,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1212,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":992}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"49'1266","reported_seq":1292,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.269488+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:15:58.269488+0000","last_peered":"2024-01-11T10:15:58.269488+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:15:58.269488+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1266,"log_dups_size":0,"ondisk_log_size":1266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67436568,"num_objects":1030,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1266,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":1030}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"49'1172","reported_seq":1202,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.054238+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:15:57.054238+0000","last_peered":"2024-01-11T10:15:57.054238+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:15:57.054238+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1172,"log_dups_size":0,"ondisk_log_size":1172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62586880,"num_objects":955,"num_object_clones":0,"num_object_copies":2865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":955,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1173,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":955}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"49'1163","reported_seq":1190,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-01-11T10:15:58.278445+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:15:58.278445+0000","last_peered":"2024-01-11T10:15:58.278445+0000","last_clean":"2024-01-11T10:15:58.278445+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:15:58.278445+0000","last_undegraded":"2024-01-11T10:15:58.278445+0000","last_fullsized":"2024-01-11T10:15:58.278445+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1163,"log_dups_size":0,"ondisk_log_size":1163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60096512,"num_objects":917,"num_object_clones":0,"num_object_copies":2751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1163,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"49'1212","reported_seq":1242,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.154961+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:15:57.154961+0000","last_peered":"2024-01-11T10:15:57.154961+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:15:57.154961+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":983,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1213,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":983}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"50'1236","reported_seq":1262,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.087943+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:15:57.087943+0000","last_peered":"2024-01-11T10:15:57.087943+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:15:57.087943+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65732608,"num_objects":1003,"num_object_clones":0,"num_object_copies":3009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1003,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1237,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1003}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"49'1158","reported_seq":1182,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.134998+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:15:58.134998+0000","last_peered":"2024-01-11T10:15:58.134998+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:15:58.134998+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61669376,"num_objects":941,"num_object_clones":0,"num_object_copies":2823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":941,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1159,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":941}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"49'1261","reported_seq":1285,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.149066+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:15:58.149066+0000","last_peered":"2024-01-11T10:15:58.149066+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:15:58.149066+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1261,"log_dups_size":0,"ondisk_log_size":1261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67239936,"num_objects":1026,"num_object_clones":0,"num_object_copies":3078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1262,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1026}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":213,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:37.251981+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:15:37.251981+0000","last_peered":"2024-01-11T10:15:37.251981+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:15:37.251981+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":70,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:56.289524+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:15:56.289524+0000","last_peered":"2024-01-11T10:15:56.289524+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:15:56.289524+0000","last_undegraded":"2024-01-11T10:15:56.289524+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"49'1185","reported_seq":1210,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.279117+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:15:58.279117+0000","last_peered":"2024-01-11T10:15:58.279117+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:15:58.279117+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1185,"log_dups_size":0,"ondisk_log_size":1185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62193664,"num_objects":949,"num_object_clones":0,"num_object_copies":2847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":949,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1185,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":949}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":66,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.361984+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:15:37.361984+0000","last_peered":"2024-01-11T10:15:37.361984+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:15:37.361984+0000","last_undegraded":"2024-01-11T10:15:37.361984+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"49'1178","reported_seq":1208,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.032070+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:15:57.032070+0000","last_peered":"2024-01-11T10:15:57.032070+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:15:57.032070+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1178,"log_dups_size":0,"ondisk_log_size":1178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63766528,"num_objects":973,"num_object_clones":0,"num_object_copies":2919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":973,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1179,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":973}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:56.276591+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:15:56.276591+0000","last_peered":"2024-01-11T10:15:56.276591+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:15:56.276591+0000","last_undegraded":"2024-01-11T10:15:56.276591+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"49'1146","reported_seq":1176,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:56.968830+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:15:56.968830+0000","last_peered":"2024-01-11T10:15:56.968830+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:15:56.968830+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1146,"log_dups_size":0,"ondisk_log_size":1146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63111168,"num_objects":963,"num_object_clones":0,"num_object_copies":2889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":963,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1147,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":963}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":68,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:58.268601+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:15:58.268601+0000","last_peered":"2024-01-11T10:15:58.268601+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:15:58.268601+0000","last_undegraded":"2024-01-11T10:15:58.268601+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"49'1251","reported_seq":1281,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:56.975683+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:15:56.975683+0000","last_peered":"2024-01-11T10:15:56.975683+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:15:56.975683+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1251,"log_dups_size":0,"ondisk_log_size":1251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67502080,"num_objects":1030,"num_object_clones":0,"num_object_copies":3090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1030,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1252,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":1030}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.243016+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:15:37.243016+0000","last_peered":"2024-01-11T10:15:37.243016+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:15:37.243016+0000","last_undegraded":"2024-01-11T10:15:37.243016+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"49'1135","reported_seq":1162,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-01-11T10:15:58.272912+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:15:58.272912+0000","last_peered":"2024-01-11T10:15:58.272912+0000","last_clean":"2024-01-11T10:15:58.272912+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:15:58.272912+0000","last_undegraded":"2024-01-11T10:15:58.272912+0000","last_fullsized":"2024-01-11T10:15:58.272912+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1135,"log_dups_size":0,"ondisk_log_size":1135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59310080,"num_objects":905,"num_object_clones":0,"num_object_copies":2715,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1135,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":82,"reported_epoch":49,"state":"active+undersized","last_fresh":"2024-01-11T10:15:37.252838+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:15:37.252838+0000","last_peered":"2024-01-11T10:15:37.252838+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:15:37.252838+0000","last_undegraded":"2024-01-11T10:15:37.252838+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"49'1201","reported_seq":1231,"reported_epoch":49,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:57.087386+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:15:57.087386+0000","last_peered":"2024-01-11T10:15:57.087386+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:15:57.087386+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1201,"log_dups_size":0,"ondisk_log_size":1201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63963136,"num_objects":976,"num_object_clones":0,"num_object_copies":2928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1202,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":976}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:58.275316+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:15:58.275316+0000","last_peered":"2024-01-11T10:15:58.275316+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:15:58.275316+0000","last_undegraded":"2024-01-11T10:15:58.275316+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"49'1214","reported_seq":1239,"reported_epoch":50,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:15:58.277406+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:15:58.277406+0000","last_peered":"2024-01-11T10:15:58.277406+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:15:58.277406+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1214,"log_dups_size":0,"ondisk_log_size":1214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65011712,"num_objects":992,"num_object_clones":0,"num_object_copies":2976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":992,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1214,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":992}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":70,"reported_epoch":50,"state":"active+undersized","last_fresh":"2024-01-11T10:15:56.299105+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:15:56.299105+0000","last_peered":"2024-01-11T10:15:56.299105+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:15:56.299105+0000","last_undegraded":"2024-01-11T10:15:56.299105+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":1025769496,"num_objects":15653,"num_object_clones":0,"num_object_copies":46959,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":19295,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19284,"ondisk_log_size":19284,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706124,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166864,"kb_used_data":51208,"kb_used_omap":0,"kb_used_meta":115648,"kb_avail":94204976,"statfs":{"total":96636764160,"available":96465895424,"internally_reserved":0,"allocated":52436992,"data_stored":832062050,"data_compressed":39505687,"data_compressed_allocated":51998720,"data_compressed_original":831897600,"omap_allocated":0,"internal_metadata":118423552},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":165112,"kb_used_data":43888,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94206728,"statfs":{"total":96636764160,"available":96467689472,"internally_reserved":0,"allocated":44941312,"data_stored":712131170,"data_compressed":33810726,"data_compressed_allocated":44503040,"data_compressed_original":711966720,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706123,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":151100,"kb_used_data":39600,"kb_used_omap":0,"kb_used_meta":111488,"kb_avail":94220740,"statfs":{"total":96636764160,"available":96482037760,"internally_reserved":0,"allocated":40550400,"data_stored":642206935,"data_compressed":30488263,"data_compressed_allocated":40128512,"data_compressed_original":642056192,"omap_allocated":0,"internal_metadata":114163712},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:16:00.008 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:16:00.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:00.009 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:00.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:00.074 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:00.092 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:00.124 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.137+0000 7ff345f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.146+0000 7ff345f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:00.266 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:00.267 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:00.269 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:00.270 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:00.270 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:00.445 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.458+0000 7f50bb26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.449 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.466+0000 7f50bb26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.471 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:00.472 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:00.475 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:00.475 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:00.590 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.603+0000 7f5c9ff1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.593 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.611+0000 7f5c9ff1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.597 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:00.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:00.601 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:00.601 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:00.723 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.733+0000 7fb7dca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.727 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:00.745+0000 7fb7dca2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:00.746 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:00.747 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:00.747 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:00.747 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:00.747 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:00.749 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:02.094 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:02.210 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:02.226+0000 7f7d91cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:02.214 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:02.232+0000 7f7d91cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:02.484 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:02.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:03.817 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~220s 2024-01-11T10:16:04.507 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:04.623 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:04.639+0000 7f7eb439e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:04.627 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:04.645+0000 7f7eb439e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:04.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:05.750 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:05.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:05.750 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:05.750 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:05.901 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:05.913+0000 7fc40355e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:05.904 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:05.922+0000 7fc40355e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.002 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:06.002 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:06.003 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:06.006 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:06.006 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:06.133 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.146+0000 7f1d6e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.137 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.155+0000 7f1d6e8ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:06.165 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:06.168 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:06.169 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:06.286 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.299+0000 7fbd19104640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.290 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.307+0000 7fbd19104640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.294 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:06.299 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:06.299 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:06.415 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.428+0000 7f6b6a4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.419 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.437+0000 7f6b6a4de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:06.434 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:06.435 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:06.437 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:06.438 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:06.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:06.922 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.939+0000 7f49a226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:06.926 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:06.944+0000 7f49a226d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:07.033 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:07.063 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:09.063 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:09.181 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:09.197+0000 7f1a310ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:09.184 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:09.202+0000 7f1a310ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:09.317 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:09.321 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~226s 2024-01-11T10:16:09.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:09.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:11.356 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:11.439 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:11.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-01-11T10:16:11.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:11.506+0000 7f17a735e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:11.493 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:11.511+0000 7f17a735e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:11.561 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:11.573+0000 7f207dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:11.564 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:11.582+0000 7f207dc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:11.627 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:11.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:12.005 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.0 to 1 (10000) 2024-01-11T10:16:12.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-01-11T10:16:12.136 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:12.148+0000 7f6ba9d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:12.139 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:12.157+0000 7f6ba9d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:13.010 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.1 to 1 (10000) 2024-01-11T10:16:13.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-01-11T10:16:13.146 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:13.158+0000 7f7bb08de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:13.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:13.169+0000 7f7bb08de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:13.658 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:13.774 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:13.790+0000 7f3ecd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:13.777 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:13.795+0000 7f3ecd86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:13.894 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:13.912 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:13.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:14.018 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.2 to 1 (10000) 2024-01-11T10:16:14.032 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-01-11T10:16:14.147 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:14.159+0000 7feed6aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:14.151 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:14.169+0000 7feed6aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:14.826 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~231s 2024-01-11T10:16:15.026 INFO:teuthology.orchestra.run.smithi040.stderr:reweighted osd.3 to 1 (10000) 2024-01-11T10:16:15.039 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-01-11T10:16:15.039 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:16:15.159 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:15.172+0000 7f1111f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:15.161 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:15.179+0000 7f1111f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:15.485 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:16:15.485 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:16:15.498 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-01-11T10:16:15.047282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":821822008,"num_objects":12534,"num_object_clones":0,"num_object_copies":37602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":22553,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22497,"ondisk_log_size":22497,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":560256,"kb_used_data":121184,"kb_used_omap":0,"kb_used_meta":439040,"kb_avail":376927104,"statfs":{"total":386547056640,"available":385973354496,"internally_reserved":0,"allocated":124092416,"data_stored":1959369778,"data_compressed":93019743,"data_compressed_allocated":122437632,"data_compressed_original":1958756352,"omap_allocated":0,"internal_metadata":449576960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67960832,"num_objects":-1037,"num_object_clones":0,"num_object_copies":-3111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.853231"},"pg_stats":[{"pgid":"5.f","version":"53'1498","reported_seq":1528,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.146146+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:16:13.146146+0000","last_peered":"2024-01-11T10:16:13.146146+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:16:13.146146+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":815}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"53'1427","reported_seq":1457,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.148672+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:16:13.148672+0000","last_peered":"2024-01-11T10:16:13.148672+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:16:13.148672+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":776}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"53'1468","reported_seq":1499,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.207393+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:16:13.207393+0000","last_peered":"2024-01-11T10:16:13.207393+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:16:13.207393+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54132760,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1469,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"53'1341","reported_seq":1377,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.053073+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:16:12.053073+0000","last_peered":"2024-01-11T10:16:12.053073+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:16:12.053073+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"53'1340","reported_seq":1372,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-01-11T10:16:13.117759+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:16:13.117759+0000","last_peered":"2024-01-11T10:16:13.117759+0000","last_clean":"2024-01-11T10:16:13.117759+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:16:13.117759+0000","last_undegraded":"2024-01-11T10:16:13.117759+0000","last_fullsized":"2024-01-11T10:16:13.117759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1340,"log_dups_size":0,"ondisk_log_size":1340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":2217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"53'1382","reported_seq":1418,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.984819+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:16:11.984819+0000","last_peered":"2024-01-11T10:16:11.984819+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:16:11.984819+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1383,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"54'1428","reported_seq":1460,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.181439+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:16:12.181439+0000","last_peered":"2024-01-11T10:16:12.181439+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:16:12.181439+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1428,"log_dups_size":0,"ondisk_log_size":1428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1429,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"53'1309","reported_seq":1339,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.209563+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:16:13.209563+0000","last_peered":"2024-01-11T10:16:13.209563+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:16:13.209563+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1310,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"53'1474","reported_seq":1504,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.079188+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:16:13.079188+0000","last_peered":"2024-01-11T10:16:13.079188+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:16:13.079188+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":219,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:00.220523+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:00.220523+0000","last_peered":"2024-01-11T10:16:00.220523+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:00.220523+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.070487+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:12.070487+0000","last_peered":"2024-01-11T10:16:12.070487+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:12.070487+0000","last_undegraded":"2024-01-11T10:16:12.070487+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"53'1385","reported_seq":1415,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.129931+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:16:13.129931+0000","last_peered":"2024-01-11T10:16:13.129931+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:16:13.129931+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49020928,"num_objects":748,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1386,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.389287+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:15:59.389287+0000","last_peered":"2024-01-11T10:15:59.389287+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:15:59.389287+0000","last_undegraded":"2024-01-11T10:15:59.389287+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"53'1369","reported_seq":1405,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.213946+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:16:12.213946+0000","last_peered":"2024-01-11T10:16:12.213946+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:16:12.213946+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51249152,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.074795+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:12.074795+0000","last_peered":"2024-01-11T10:16:12.074795+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:12.074795+0000","last_undegraded":"2024-01-11T10:16:12.074795+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"53'1338","reported_seq":1374,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.794437+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:16:11.794437+0000","last_peered":"2024-01-11T10:16:11.794437+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:16:11.794437+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":2313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1339,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.355962+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:15:59.355962+0000","last_peered":"2024-01-11T10:15:59.355962+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:15:59.355962+0000","last_undegraded":"2024-01-11T10:15:59.355962+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"53'1470","reported_seq":1506,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.960205+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:16:11.960205+0000","last_peered":"2024-01-11T10:16:11.960205+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:16:11.960205+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1470,"log_dups_size":0,"ondisk_log_size":1470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1471,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:16:00.225413+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:00.225413+0000","last_peered":"2024-01-11T10:16:00.225413+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:00.225413+0000","last_undegraded":"2024-01-11T10:16:00.225413+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"53'1330","reported_seq":1362,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-01-11T10:16:13.156987+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:16:13.156987+0000","last_peered":"2024-01-11T10:16:13.156987+0000","last_clean":"2024-01-11T10:16:13.156987+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:16:13.156987+0000","last_undegraded":"2024-01-11T10:16:13.156987+0000","last_fullsized":"2024-01-11T10:16:13.156987+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46465024,"num_objects":709,"num_object_clones":0,"num_object_copies":2127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:16:00.228926+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:00.228926+0000","last_peered":"2024-01-11T10:16:00.228926+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:00.228926+0000","last_undegraded":"2024-01-11T10:16:00.228926+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"53'1421","reported_seq":1457,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.979033+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:16:11.979033+0000","last_peered":"2024-01-11T10:16:11.979033+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:16:11.979033+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1422,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.351629+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:15:59.351629+0000","last_peered":"2024-01-11T10:15:59.351629+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:15:59.351629+0000","last_undegraded":"2024-01-11T10:15:59.351629+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"53'1420","reported_seq":1450,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.246451+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:16:13.246451+0000","last_peered":"2024-01-11T10:16:13.246451+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:16:13.246451+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":785}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.079104+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:12.079104+0000","last_peered":"2024-01-11T10:16:12.079104+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:12.079104+0000","last_undegraded":"2024-01-11T10:16:12.079104+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":821231640,"num_objects":12532,"num_object_clones":0,"num_object_copies":37596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22416,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22400,"ondisk_log_size":22400,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166420,"kb_used_data":41036,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94205420,"statfs":{"total":96636764160,"available":96466350080,"internally_reserved":0,"allocated":42020864,"data_stored":664892694,"data_compressed":31566975,"data_compressed_allocated":41549824,"data_compressed_original":664715264,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164924,"kb_used_data":35380,"kb_used_omap":0,"kb_used_meta":129536,"kb_avail":94206916,"statfs":{"total":96636764160,"available":96467881984,"internally_reserved":0,"allocated":36229120,"data_stored":572224790,"data_compressed":27166606,"data_compressed_allocated":35758080,"data_compressed_original":572047360,"omap_allocated":0,"internal_metadata":132644864},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153136,"kb_used_data":31912,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94218704,"statfs":{"total":96636764160,"available":96479952896,"internally_reserved":0,"allocated":32677888,"data_stored":515863262,"data_compressed":24488327,"data_compressed_allocated":32231424,"data_compressed_original":515702784,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:16:15.499 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:16:15.618 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:15.631+0000 7f235539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:15.622 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:15.640+0000 7f235539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:15.822 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:16:15.823 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:16:15.835 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-01-11T10:16:15.047282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":821822008,"num_objects":12534,"num_object_clones":0,"num_object_copies":37602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":22553,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22497,"ondisk_log_size":22497,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":560256,"kb_used_data":121184,"kb_used_omap":0,"kb_used_meta":439040,"kb_avail":376927104,"statfs":{"total":386547056640,"available":385973354496,"internally_reserved":0,"allocated":124092416,"data_stored":1959369778,"data_compressed":93019743,"data_compressed_allocated":122437632,"data_compressed_original":1958756352,"omap_allocated":0,"internal_metadata":449576960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67960832,"num_objects":-1037,"num_object_clones":0,"num_object_copies":-3111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.853231"},"pg_stats":[{"pgid":"5.f","version":"53'1498","reported_seq":1528,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.146146+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:16:13.146146+0000","last_peered":"2024-01-11T10:16:13.146146+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:16:13.146146+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":815}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"53'1427","reported_seq":1457,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.148672+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:16:13.148672+0000","last_peered":"2024-01-11T10:16:13.148672+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:16:13.148672+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":776}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"53'1468","reported_seq":1499,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.207393+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:16:13.207393+0000","last_peered":"2024-01-11T10:16:13.207393+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:16:13.207393+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54132760,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1469,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"53'1341","reported_seq":1377,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.053073+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:16:12.053073+0000","last_peered":"2024-01-11T10:16:12.053073+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:16:12.053073+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"53'1340","reported_seq":1372,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-01-11T10:16:13.117759+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:16:13.117759+0000","last_peered":"2024-01-11T10:16:13.117759+0000","last_clean":"2024-01-11T10:16:13.117759+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:16:13.117759+0000","last_undegraded":"2024-01-11T10:16:13.117759+0000","last_fullsized":"2024-01-11T10:16:13.117759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1340,"log_dups_size":0,"ondisk_log_size":1340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":2217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"53'1382","reported_seq":1418,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.984819+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:16:11.984819+0000","last_peered":"2024-01-11T10:16:11.984819+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:16:11.984819+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1383,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"54'1428","reported_seq":1460,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.181439+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:16:12.181439+0000","last_peered":"2024-01-11T10:16:12.181439+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:16:12.181439+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1428,"log_dups_size":0,"ondisk_log_size":1428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1429,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"53'1309","reported_seq":1339,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.209563+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:16:13.209563+0000","last_peered":"2024-01-11T10:16:13.209563+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:16:13.209563+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1310,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"53'1474","reported_seq":1504,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.079188+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:16:13.079188+0000","last_peered":"2024-01-11T10:16:13.079188+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:16:13.079188+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":219,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:00.220523+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:00.220523+0000","last_peered":"2024-01-11T10:16:00.220523+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:00.220523+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.070487+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:12.070487+0000","last_peered":"2024-01-11T10:16:12.070487+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:12.070487+0000","last_undegraded":"2024-01-11T10:16:12.070487+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"53'1385","reported_seq":1415,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.129931+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:16:13.129931+0000","last_peered":"2024-01-11T10:16:13.129931+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:16:13.129931+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49020928,"num_objects":748,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1386,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.389287+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:15:59.389287+0000","last_peered":"2024-01-11T10:15:59.389287+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:15:59.389287+0000","last_undegraded":"2024-01-11T10:15:59.389287+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"53'1369","reported_seq":1405,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.213946+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:16:12.213946+0000","last_peered":"2024-01-11T10:16:12.213946+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:16:12.213946+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51249152,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.074795+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:12.074795+0000","last_peered":"2024-01-11T10:16:12.074795+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:12.074795+0000","last_undegraded":"2024-01-11T10:16:12.074795+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"53'1338","reported_seq":1374,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.794437+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:16:11.794437+0000","last_peered":"2024-01-11T10:16:11.794437+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:16:11.794437+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":2313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1339,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.355962+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:15:59.355962+0000","last_peered":"2024-01-11T10:15:59.355962+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:15:59.355962+0000","last_undegraded":"2024-01-11T10:15:59.355962+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"53'1470","reported_seq":1506,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.960205+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:16:11.960205+0000","last_peered":"2024-01-11T10:16:11.960205+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:16:11.960205+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1470,"log_dups_size":0,"ondisk_log_size":1470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1471,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:16:00.225413+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:00.225413+0000","last_peered":"2024-01-11T10:16:00.225413+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:00.225413+0000","last_undegraded":"2024-01-11T10:16:00.225413+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"53'1330","reported_seq":1362,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-01-11T10:16:13.156987+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:16:13.156987+0000","last_peered":"2024-01-11T10:16:13.156987+0000","last_clean":"2024-01-11T10:16:13.156987+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:16:13.156987+0000","last_undegraded":"2024-01-11T10:16:13.156987+0000","last_fullsized":"2024-01-11T10:16:13.156987+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46465024,"num_objects":709,"num_object_clones":0,"num_object_copies":2127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:16:00.228926+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:00.228926+0000","last_peered":"2024-01-11T10:16:00.228926+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:00.228926+0000","last_undegraded":"2024-01-11T10:16:00.228926+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"53'1421","reported_seq":1457,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.979033+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:16:11.979033+0000","last_peered":"2024-01-11T10:16:11.979033+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:16:11.979033+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1422,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.351629+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:15:59.351629+0000","last_peered":"2024-01-11T10:15:59.351629+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:15:59.351629+0000","last_undegraded":"2024-01-11T10:15:59.351629+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"53'1420","reported_seq":1450,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.246451+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:16:13.246451+0000","last_peered":"2024-01-11T10:16:13.246451+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:16:13.246451+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":785}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.079104+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:12.079104+0000","last_peered":"2024-01-11T10:16:12.079104+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:12.079104+0000","last_undegraded":"2024-01-11T10:16:12.079104+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":821231640,"num_objects":12532,"num_object_clones":0,"num_object_copies":37596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22416,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22400,"ondisk_log_size":22400,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166420,"kb_used_data":41036,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94205420,"statfs":{"total":96636764160,"available":96466350080,"internally_reserved":0,"allocated":42020864,"data_stored":664892694,"data_compressed":31566975,"data_compressed_allocated":41549824,"data_compressed_original":664715264,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164924,"kb_used_data":35380,"kb_used_omap":0,"kb_used_meta":129536,"kb_avail":94206916,"statfs":{"total":96636764160,"available":96467881984,"internally_reserved":0,"allocated":36229120,"data_stored":572224790,"data_compressed":27166606,"data_compressed_allocated":35758080,"data_compressed_original":572047360,"omap_allocated":0,"internal_metadata":132644864},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153136,"kb_used_data":31912,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94218704,"statfs":{"total":96636764160,"available":96479952896,"internally_reserved":0,"allocated":32677888,"data_stored":515863262,"data_compressed":24488327,"data_compressed_allocated":32231424,"data_compressed_original":515702784,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:16:15.837 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-01-11T10:16:15.837 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:15.837 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:16:15.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:15.969 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:15.983+0000 7fa20c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:15.973 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:15.991+0000 7fa20c02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:16.045 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:16.062+0000 7f7a9466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:16.049 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:16.067+0000 7f7a9466d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:16.186 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:16.210 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:16.211 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:16.245 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:16:16.246 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:16:16.263 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-01-11T10:16:16.195933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":821822008,"num_objects":12534,"num_object_clones":0,"num_object_copies":37602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11086,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":22553,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22497,"ondisk_log_size":22497,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":560256,"kb_used_data":121184,"kb_used_omap":0,"kb_used_meta":439040,"kb_avail":376927104,"statfs":{"total":386547056640,"available":385973354496,"internally_reserved":0,"allocated":124092416,"data_stored":1959369778,"data_compressed":93019743,"data_compressed_allocated":122437632,"data_compressed_original":1958756352,"omap_allocated":0,"internal_metadata":449576960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-67960832,"num_objects":-1037,"num_object_clones":0,"num_object_copies":-3111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-924,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1037,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.170523"},"pg_stats":[{"pgid":"5.f","version":"53'1498","reported_seq":1528,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.146146+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:16:13.146146+0000","last_peered":"2024-01-11T10:16:13.146146+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:16:13.146146+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1498,"log_dups_size":0,"ondisk_log_size":1498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53411840,"num_objects":815,"num_object_clones":0,"num_object_copies":2445,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":815,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1499,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":815}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"53'1427","reported_seq":1457,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.148672+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:16:13.148672+0000","last_peered":"2024-01-11T10:16:13.148672+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:16:13.148672+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1427,"log_dups_size":0,"ondisk_log_size":1427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50855936,"num_objects":776,"num_object_clones":0,"num_object_copies":2328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":776,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1428,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":776}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"53'1468","reported_seq":1499,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.207393+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:16:13.207393+0000","last_peered":"2024-01-11T10:16:13.207393+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:16:13.207393+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1468,"log_dups_size":0,"ondisk_log_size":1468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54132760,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":827,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1469,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":827}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"53'1341","reported_seq":1377,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.053073+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:16:12.053073+0000","last_peered":"2024-01-11T10:16:12.053073+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:16:12.053073+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1341,"log_dups_size":0,"ondisk_log_size":1341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51511296,"num_objects":786,"num_object_clones":0,"num_object_copies":2358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":786,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1342,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":786}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"53'1340","reported_seq":1372,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-01-11T10:16:13.117759+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:16:13.117759+0000","last_peered":"2024-01-11T10:16:13.117759+0000","last_clean":"2024-01-11T10:16:13.117759+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:16:13.117759+0000","last_undegraded":"2024-01-11T10:16:13.117759+0000","last_fullsized":"2024-01-11T10:16:13.117759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1340,"log_dups_size":0,"ondisk_log_size":1340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48431104,"num_objects":739,"num_object_clones":0,"num_object_copies":2217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"53'1382","reported_seq":1418,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.984819+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:16:11.984819+0000","last_peered":"2024-01-11T10:16:11.984819+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:16:11.984819+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1382,"log_dups_size":0,"ondisk_log_size":1382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1383,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":813}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"54'1428","reported_seq":1460,"reported_epoch":54,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.181439+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:16:12.181439+0000","last_peered":"2024-01-11T10:16:12.181439+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:16:12.181439+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1428,"log_dups_size":0,"ondisk_log_size":1428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1429,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":811}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"53'1309","reported_seq":1339,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.209563+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:16:13.209563+0000","last_peered":"2024-01-11T10:16:13.209563+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:16:13.209563+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51773440,"num_objects":790,"num_object_clones":0,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":790,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1310,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":790}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"53'1474","reported_seq":1504,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.079188+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:16:13.079188+0000","last_peered":"2024-01-11T10:16:13.079188+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:16:13.079188+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1474,"log_dups_size":0,"ondisk_log_size":1474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53280768,"num_objects":813,"num_object_clones":0,"num_object_copies":2439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1475,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":813}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":219,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:00.220523+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:00.220523+0000","last_peered":"2024-01-11T10:16:00.220523+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:00.220523+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.070487+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:12.070487+0000","last_peered":"2024-01-11T10:16:12.070487+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:12.070487+0000","last_undegraded":"2024-01-11T10:16:12.070487+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"53'1385","reported_seq":1415,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.129931+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:16:13.129931+0000","last_peered":"2024-01-11T10:16:13.129931+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:16:13.129931+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1385,"log_dups_size":0,"ondisk_log_size":1385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49020928,"num_objects":748,"num_object_clones":0,"num_object_copies":2244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":748,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1386,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":748}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.389287+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:15:59.389287+0000","last_peered":"2024-01-11T10:15:59.389287+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:15:59.389287+0000","last_undegraded":"2024-01-11T10:15:59.389287+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"53'1369","reported_seq":1405,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:12.213946+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:16:12.213946+0000","last_peered":"2024-01-11T10:16:12.213946+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:16:12.213946+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1369,"log_dups_size":0,"ondisk_log_size":1369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51249152,"num_objects":782,"num_object_clones":0,"num_object_copies":2346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":782,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1370,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":782}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.074795+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:12.074795+0000","last_peered":"2024-01-11T10:16:12.074795+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:12.074795+0000","last_undegraded":"2024-01-11T10:16:12.074795+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"53'1338","reported_seq":1374,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.794437+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:16:11.794437+0000","last_peered":"2024-01-11T10:16:11.794437+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:16:11.794437+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1338,"log_dups_size":0,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50528256,"num_objects":771,"num_object_clones":0,"num_object_copies":2313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":771,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1339,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":771}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":72,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.355962+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:15:59.355962+0000","last_peered":"2024-01-11T10:15:59.355962+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:15:59.355962+0000","last_undegraded":"2024-01-11T10:15:59.355962+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"53'1470","reported_seq":1506,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.960205+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:16:11.960205+0000","last_peered":"2024-01-11T10:16:11.960205+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:16:11.960205+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1470,"log_dups_size":0,"ondisk_log_size":1470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53149696,"num_objects":811,"num_object_clones":0,"num_object_copies":2433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":811,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1471,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":811}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:16:00.225413+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:00.225413+0000","last_peered":"2024-01-11T10:16:00.225413+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:00.225413+0000","last_undegraded":"2024-01-11T10:16:00.225413+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"53'1330","reported_seq":1362,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-01-11T10:16:13.156987+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:16:13.156987+0000","last_peered":"2024-01-11T10:16:13.156987+0000","last_clean":"2024-01-11T10:16:13.156987+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:16:13.156987+0000","last_undegraded":"2024-01-11T10:16:13.156987+0000","last_fullsized":"2024-01-11T10:16:13.156987+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46465024,"num_objects":709,"num_object_clones":0,"num_object_copies":2127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:16:00.228926+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:00.228926+0000","last_peered":"2024-01-11T10:16:00.228926+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:00.228926+0000","last_undegraded":"2024-01-11T10:16:00.228926+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"53'1421","reported_seq":1457,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:11.979033+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:16:11.979033+0000","last_peered":"2024-01-11T10:16:11.979033+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:16:11.979033+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1421,"log_dups_size":0,"ondisk_log_size":1421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":756,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1422,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":756}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":53,"state":"active+undersized","last_fresh":"2024-01-11T10:15:59.351629+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:15:59.351629+0000","last_peered":"2024-01-11T10:15:59.351629+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:15:59.351629+0000","last_undegraded":"2024-01-11T10:15:59.351629+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"53'1420","reported_seq":1450,"reported_epoch":53,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:13.246451+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:16:13.246451+0000","last_peered":"2024-01-11T10:16:13.246451+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:16:13.246451+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1420,"log_dups_size":0,"ondisk_log_size":1420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51445760,"num_objects":785,"num_object_clones":0,"num_object_copies":2355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":785,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1421,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":785}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":54,"state":"active+undersized","last_fresh":"2024-01-11T10:16:12.079104+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:12.079104+0000","last_peered":"2024-01-11T10:16:12.079104+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:12.079104+0000","last_undegraded":"2024-01-11T10:16:12.079104+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":821231640,"num_objects":12532,"num_object_clones":0,"num_object_copies":37596,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11084,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":22416,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22400,"ondisk_log_size":22400,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706139,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":166420,"kb_used_data":41036,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94205420,"statfs":{"total":96636764160,"available":96466350080,"internally_reserved":0,"allocated":42020864,"data_stored":664892694,"data_compressed":31566975,"data_compressed_allocated":41549824,"data_compressed_original":664715264,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":164924,"kb_used_data":35380,"kb_used_omap":0,"kb_used_meta":129536,"kb_avail":94206916,"statfs":{"total":96636764160,"available":96467881984,"internally_reserved":0,"allocated":36229120,"data_stored":572224790,"data_compressed":27166606,"data_compressed_allocated":35758080,"data_compressed_original":572047360,"omap_allocated":0,"internal_metadata":132644864},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706138,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":153136,"kb_used_data":31912,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94218704,"statfs":{"total":96636764160,"available":96479952896,"internally_reserved":0,"allocated":32677888,"data_stored":515863262,"data_compressed":24488327,"data_compressed_allocated":32231424,"data_compressed_original":515702784,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:16:18.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:18.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:18.340+0000 7f7b5886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:18.328 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:18.346+0000 7f7b5886d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:18.438 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:18.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:20.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~237s 2024-01-11T10:16:20.461 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:20.585 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:20.601+0000 7fd6e206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:20.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:20.606+0000 7fd6e206d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:20.716 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:20.734 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:21.265 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:21.265 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:21.265 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:21.265 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:21.382 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.395+0000 7f0b5699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.385 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.403+0000 7f0b5699e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.501 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:21.502 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:21.502 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:21.502 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:21.502 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:21.502 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:21.503 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:21.504 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:21.505 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:21.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:21.626 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.637+0000 7f39dcd04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.629 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.647+0000 7f39dcd04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.678 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:21.678 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:21.678 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:21.678 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:21.679 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:21.682 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:21.682 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:21.817 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.830+0000 7fcaf491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.839+0000 7fcaf491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.825 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:21.829 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:21.829 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:21.829 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:21.963 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.976+0000 7f6831d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.967 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:21.985+0000 7f6831d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:21.984 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:21.984 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:21.985 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:21.988 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:22.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:22.862 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:22.878+0000 7f433c51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:22.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:22.883+0000 7f433c51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:23.080 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:23.102 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:23.102 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:25.103 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:25.219 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:25.235+0000 7fa9b951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:25.222 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:25.240+0000 7fa9b951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:25.397 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:25.422 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:25.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~242s 2024-01-11T10:16:26.989 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:26.989 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:26.989 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:16:27.126 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:27.140+0000 7fab6c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:27.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:27.148+0000 7fab6c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:27.424 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:27.425 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:16:27.426 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:16:27.437 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-01-11T10:16:26.198543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":693371448,"num_objects":10574,"num_object_clones":0,"num_object_copies":31722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9324,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":24513,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24457,"ondisk_log_size":24457,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":543896,"kb_used_data":104816,"kb_used_omap":0,"kb_used_meta":439040,"kb_avail":376943464,"statfs":{"total":386547056640,"available":385990107136,"internally_reserved":0,"allocated":107331584,"data_stored":1689662542,"data_compressed":80210751,"data_compressed_allocated":105578496,"data_compressed_original":1689010176,"omap_allocated":0,"internal_metadata":449576960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-128450560,"num_objects":-1960,"num_object_clones":0,"num_object_copies":-5880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1960,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.151261"},"pg_stats":[{"pgid":"5.f","version":"57'1639","reported_seq":1673,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.212248+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:16:23.212248+0000","last_peered":"2024-01-11T10:16:23.212248+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:16:23.212248+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1639,"log_dups_size":0,"ondisk_log_size":1639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44171264,"num_objects":674,"num_object_clones":0,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":674,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1640,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":674}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"57'1552","reported_seq":1586,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.086458+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:16:23.086458+0000","last_peered":"2024-01-11T10:16:23.086458+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:16:23.086458+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1552,"log_dups_size":0,"ondisk_log_size":1552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42663936,"num_objects":651,"num_object_clones":0,"num_object_copies":1953,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":651,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1553,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":651}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"57'1571","reported_seq":1606,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.003587+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:16:23.003587+0000","last_peered":"2024-01-11T10:16:23.003587+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:16:23.003587+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1571,"log_dups_size":0,"ondisk_log_size":1571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47382552,"num_objects":724,"num_object_clones":0,"num_object_copies":2172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1572,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":724}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"57'1466","reported_seq":1506,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.065984+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:16:22.065984+0000","last_peered":"2024-01-11T10:16:22.065984+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:16:22.065984+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43319296,"num_objects":661,"num_object_clones":0,"num_object_copies":1983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1467,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":661}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"57'1436","reported_seq":1472,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:16:22.799678+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:16:22.799678+0000","last_peered":"2024-01-11T10:16:22.799678+0000","last_clean":"2024-01-11T10:16:22.799678+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:16:22.799678+0000","last_undegraded":"2024-01-11T10:16:22.799678+0000","last_fullsized":"2024-01-11T10:16:22.799678+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1436,"log_dups_size":0,"ondisk_log_size":1436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42139648,"num_objects":643,"num_object_clones":0,"num_object_copies":1929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1437,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"57'1511","reported_seq":1551,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.128249+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:16:22.128249+0000","last_peered":"2024-01-11T10:16:22.128249+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:16:22.128249+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44826624,"num_objects":684,"num_object_clones":0,"num_object_copies":2052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":684,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1512,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":684}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"57'1563","reported_seq":1597,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.019553+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:16:22.019553+0000","last_peered":"2024-01-11T10:16:22.019553+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:16:22.019553+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44302336,"num_objects":676,"num_object_clones":0,"num_object_copies":2028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":676,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1564,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":676}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"57'1443","reported_seq":1477,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.182026+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:16:23.182026+0000","last_peered":"2024-01-11T10:16:23.182026+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:16:23.182026+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1443,"log_dups_size":0,"ondisk_log_size":1443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42991616,"num_objects":656,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1444,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":656}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"57'1616","reported_seq":1650,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.158108+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:16:23.158108+0000","last_peered":"2024-01-11T10:16:23.158108+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:16:23.158108+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43974656,"num_objects":671,"num_object_clones":0,"num_object_copies":2013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":671,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1617,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":671}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":223,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:15.097279+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:15.097279+0000","last_peered":"2024-01-11T10:16:15.097279+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:15.097279+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.344665+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:16.344665+0000","last_peered":"2024-01-11T10:16:16.344665+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:16.344665+0000","last_undegraded":"2024-01-11T10:16:16.344665+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"57'1505","reported_seq":1539,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.132730+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:16:23.132730+0000","last_peered":"2024-01-11T10:16:23.132730+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:16:23.132730+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1505,"log_dups_size":0,"ondisk_log_size":1505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":628,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1506,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":628}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.303007+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:16:16.303007+0000","last_peered":"2024-01-11T10:16:16.303007+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:16:16.303007+0000","last_undegraded":"2024-01-11T10:16:16.303007+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"57'1495","reported_seq":1535,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.196971+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:16:22.196971+0000","last_peered":"2024-01-11T10:16:22.196971+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:16:22.196971+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1495,"log_dups_size":0,"ondisk_log_size":1495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42991616,"num_objects":656,"num_object_clones":0,"num_object_copies":1968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":656,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1496,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":656}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.386001+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:16.386001+0000","last_peered":"2024-01-11T10:16:16.386001+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:16.386001+0000","last_undegraded":"2024-01-11T10:16:16.386001+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"57'1452","reported_seq":1492,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.013136+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:16:22.013136+0000","last_peered":"2024-01-11T10:16:22.013136+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:16:22.013136+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1452,"log_dups_size":0,"ondisk_log_size":1452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43057152,"num_objects":657,"num_object_clones":0,"num_object_copies":1971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1453,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":657}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.292134+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:16:16.292134+0000","last_peered":"2024-01-11T10:16:16.292134+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:16:16.292134+0000","last_undegraded":"2024-01-11T10:16:16.292134+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"57'1596","reported_seq":1636,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.184467+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:16:22.184467+0000","last_peered":"2024-01-11T10:16:22.184467+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:16:22.184467+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1596,"log_dups_size":0,"ondisk_log_size":1596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":685,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1597,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":685}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.093416+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:15.093416+0000","last_peered":"2024-01-11T10:16:15.093416+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:15.093416+0000","last_undegraded":"2024-01-11T10:16:15.093416+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"57'1432","reported_seq":1468,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:16:23.251403+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:16:23.251403+0000","last_peered":"2024-01-11T10:16:23.251403+0000","last_clean":"2024-01-11T10:16:23.251403+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:16:23.251403+0000","last_undegraded":"2024-01-11T10:16:23.251403+0000","last_fullsized":"2024-01-11T10:16:23.251403+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1432,"log_dups_size":0,"ondisk_log_size":1432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39780352,"num_objects":607,"num_object_clones":0,"num_object_copies":1821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1433,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.080452+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:15.080452+0000","last_peered":"2024-01-11T10:16:15.080452+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:15.080452+0000","last_undegraded":"2024-01-11T10:16:15.080452+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"57'1530","reported_seq":1570,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:22.086483+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:16:22.086483+0000","last_peered":"2024-01-11T10:16:22.086483+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:16:22.086483+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1530,"log_dups_size":0,"ondisk_log_size":1530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42401792,"num_objects":647,"num_object_clones":0,"num_object_copies":1941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":647,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1531,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":647}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.320111+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:16:16.320111+0000","last_peered":"2024-01-11T10:16:16.320111+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:16:16.320111+0000","last_undegraded":"2024-01-11T10:16:16.320111+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"57'1553","reported_seq":1587,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:23.062143+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:16:23.062143+0000","last_peered":"2024-01-11T10:16:23.062143+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:16:23.062143+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1553,"log_dups_size":0,"ondisk_log_size":1553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42729472,"num_objects":652,"num_object_clones":0,"num_object_copies":1956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1554,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":652}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.338236+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:16.338236+0000","last_peered":"2024-01-11T10:16:16.338236+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:16.338236+0000","last_undegraded":"2024-01-11T10:16:16.338236+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":692781080,"num_objects":10572,"num_object_clones":0,"num_object_copies":31716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9322,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":24376,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24360,"ondisk_log_size":24360,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":168436,"kb_used_data":34732,"kb_used_omap":0,"kb_used_meta":133696,"kb_avail":94203404,"statfs":{"total":96636764160,"available":96464285696,"internally_reserved":0,"allocated":35565568,"data_stored":561096650,"data_compressed":26637567,"data_compressed_allocated":35061760,"data_compressed_original":560906240,"omap_allocated":0,"internal_metadata":136904704},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":147276,"kb_used_data":30204,"kb_used_omap":0,"kb_used_meta":117056,"kb_avail":94224564,"statfs":{"total":96636764160,"available":96485953536,"internally_reserved":0,"allocated":30928896,"data_stored":486909898,"data_compressed":23114782,"data_compressed_allocated":30425088,"data_compressed_original":486719488,"omap_allocated":0,"internal_metadata":119865344},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706148,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152408,"kb_used_data":27024,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94219432,"statfs":{"total":96636764160,"available":96480698368,"internally_reserved":0,"allocated":27672576,"data_stored":435266962,"data_compressed":20660567,"data_compressed_allocated":27193344,"data_compressed_original":435093504,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:16:27.574 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:27.590+0000 7ff79f11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:27.578 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:27.596+0000 7ff79f11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:27.788 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:27.807 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:29.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:29.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:29.941+0000 7f03134de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:29.929 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:29.947+0000 7f03134de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:30.080 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:30.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:30.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:31.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~248s 2024-01-11T10:16:32.117 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:32.230 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:32.246+0000 7f30a7e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:32.233 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:32.251+0000 7f30a7e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:32.362 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:32.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:32.440 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:32.440 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:32.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:16:32.555 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:32.569+0000 7fc81942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:32.559 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:32.577+0000 7fc81942d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:32.865 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:16:32.865 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:16:32.882 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-01-11T10:16:32.200496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":626786872,"num_objects":9558,"num_object_clones":0,"num_object_copies":28674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":25529,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25473,"ondisk_log_size":25473,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":526904,"kb_used_data":96152,"kb_used_omap":0,"kb_used_meta":430720,"kb_avail":376960456,"statfs":{"total":386547056640,"available":386007506944,"internally_reserved":0,"allocated":98459648,"data_stored":1547711566,"data_compressed":73470159,"data_compressed_allocated":96706560,"data_compressed_original":1547059200,"omap_allocated":0,"internal_metadata":441057280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-131661824,"num_objects":-2009,"num_object_clones":0,"num_object_copies":-6027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1800,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2009,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003431"},"pg_stats":[{"pgid":"5.f","version":"57'1712","reported_seq":1746,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.044561+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:16:28.044561+0000","last_peered":"2024-01-11T10:16:28.044561+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:16:28.044561+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1712,"log_dups_size":0,"ondisk_log_size":1712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1713,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":601}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"57'1606","reported_seq":1640,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.086773+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:16:28.086773+0000","last_peered":"2024-01-11T10:16:28.086773+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:16:28.086773+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1606,"log_dups_size":0,"ondisk_log_size":1606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1607,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"57'1646","reported_seq":1681,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.178519+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:16:28.178519+0000","last_peered":"2024-01-11T10:16:28.178519+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:16:28.178519+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1646,"log_dups_size":0,"ondisk_log_size":1646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42467352,"num_objects":649,"num_object_clones":0,"num_object_copies":1947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":649,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1647,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":649}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"57'1526","reported_seq":1566,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:27.068242+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:16:27.068242+0000","last_peered":"2024-01-11T10:16:27.068242+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:16:27.068242+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1526,"log_dups_size":0,"ondisk_log_size":1526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1527,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"57'1486","reported_seq":1522,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:16:28.156335+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:16:28.156335+0000","last_peered":"2024-01-11T10:16:28.156335+0000","last_clean":"2024-01-11T10:16:28.156335+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:16:28.156335+0000","last_undegraded":"2024-01-11T10:16:28.156335+0000","last_fullsized":"2024-01-11T10:16:28.156335+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38862848,"num_objects":593,"num_object_clones":0,"num_object_copies":1779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"57'1581","reported_seq":1621,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:27.084674+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:16:27.084674+0000","last_peered":"2024-01-11T10:16:27.084674+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:16:27.084674+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40239104,"num_objects":614,"num_object_clones":0,"num_object_copies":1842,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":614,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1582,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":614}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"57'1634","reported_seq":1668,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:27.213783+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:16:27.213783+0000","last_peered":"2024-01-11T10:16:27.213783+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:16:27.213783+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1634,"log_dups_size":0,"ondisk_log_size":1634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39649280,"num_objects":605,"num_object_clones":0,"num_object_copies":1815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":605,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1635,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":605}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"57'1502","reported_seq":1536,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.206459+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:16:28.206459+0000","last_peered":"2024-01-11T10:16:28.206459+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:16:28.206459+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1502,"log_dups_size":0,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39124992,"num_objects":597,"num_object_clones":0,"num_object_copies":1791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":597,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1503,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":597}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"57'1695","reported_seq":1729,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.187088+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:16:28.187088+0000","last_peered":"2024-01-11T10:16:28.187088+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:16:28.187088+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1695,"log_dups_size":0,"ondisk_log_size":1695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":38797312,"num_objects":592,"num_object_clones":0,"num_object_copies":1776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":592,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1696,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":592}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":223,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:15.097279+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:15.097279+0000","last_peered":"2024-01-11T10:16:15.097279+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:15.097279+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.344665+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:16.344665+0000","last_peered":"2024-01-11T10:16:16.344665+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:16.344665+0000","last_undegraded":"2024-01-11T10:16:16.344665+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"57'1573","reported_seq":1607,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.188070+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:16:28.188070+0000","last_peered":"2024-01-11T10:16:28.188070+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:16:28.188070+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36700160,"num_objects":560,"num_object_clones":0,"num_object_copies":1680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":560,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1574,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":560}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.303007+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:16:16.303007+0000","last_peered":"2024-01-11T10:16:16.303007+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:16:16.303007+0000","last_undegraded":"2024-01-11T10:16:16.303007+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"57'1550","reported_seq":1590,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:27.091845+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:16:27.091845+0000","last_peered":"2024-01-11T10:16:27.091845+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:16:27.091845+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1550,"log_dups_size":0,"ondisk_log_size":1550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1551,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.386001+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:16.386001+0000","last_peered":"2024-01-11T10:16:16.386001+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:16.386001+0000","last_undegraded":"2024-01-11T10:16:16.386001+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"57'1510","reported_seq":1550,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:26.972341+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:16:26.972341+0000","last_peered":"2024-01-11T10:16:26.972341+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:16:26.972341+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1510,"log_dups_size":0,"ondisk_log_size":1510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39256064,"num_objects":599,"num_object_clones":0,"num_object_copies":1797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":599,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1511,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":599}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.292134+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:16:16.292134+0000","last_peered":"2024-01-11T10:16:16.292134+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:16:16.292134+0000","last_undegraded":"2024-01-11T10:16:16.292134+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"57'1666","reported_seq":1706,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:26.911615+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:16:26.911615+0000","last_peered":"2024-01-11T10:16:26.911615+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:16:26.911615+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1666,"log_dups_size":0,"ondisk_log_size":1666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":40304640,"num_objects":615,"num_object_clones":0,"num_object_copies":1845,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1667,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":615}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.093416+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:15.093416+0000","last_peered":"2024-01-11T10:16:15.093416+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:15.093416+0000","last_undegraded":"2024-01-11T10:16:15.093416+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"57'1486","reported_seq":1522,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:16:27.826052+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:16:27.826052+0000","last_peered":"2024-01-11T10:16:27.826052+0000","last_clean":"2024-01-11T10:16:27.826052+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:16:27.826052+0000","last_undegraded":"2024-01-11T10:16:27.826052+0000","last_fullsized":"2024-01-11T10:16:27.826052+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1486,"log_dups_size":0,"ondisk_log_size":1486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":36241408,"num_objects":553,"num_object_clones":0,"num_object_copies":1659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1487,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.080452+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:15.080452+0000","last_peered":"2024-01-11T10:16:15.080452+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:15.080452+0000","last_undegraded":"2024-01-11T10:16:15.080452+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"57'1576","reported_seq":1616,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:26.984655+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:16:26.984655+0000","last_peered":"2024-01-11T10:16:26.984655+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:16:26.984655+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":39387136,"num_objects":601,"num_object_clones":0,"num_object_copies":1803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1577,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":601}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.320111+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:16:16.320111+0000","last_peered":"2024-01-11T10:16:16.320111+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:16:16.320111+0000","last_undegraded":"2024-01-11T10:16:16.320111+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"57'1627","reported_seq":1661,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:28.141914+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:16:28.141914+0000","last_peered":"2024-01-11T10:16:28.141914+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:16:28.141914+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1627,"log_dups_size":0,"ondisk_log_size":1627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37879808,"num_objects":578,"num_object_clones":0,"num_object_copies":1734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":578,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1628,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":578}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.338236+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:16.338236+0000","last_peered":"2024-01-11T10:16:16.338236+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:16.338236+0000","last_undegraded":"2024-01-11T10:16:16.338236+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":626196504,"num_objects":9556,"num_object_clones":0,"num_object_copies":28668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8410,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":25392,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25376,"ondisk_log_size":25376,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706154,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":152588,"kb_used_data":31364,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94219252,"statfs":{"total":96636764160,"available":96480514048,"internally_reserved":0,"allocated":32116736,"data_stored":505915338,"data_compressed":24017263,"data_compressed_allocated":31612928,"data_compressed_original":505724928,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":148712,"kb_used_data":27488,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94223128,"statfs":{"total":96636764160,"available":96484483072,"internally_reserved":0,"allocated":28147712,"data_stored":442410954,"data_compressed":21001734,"data_compressed_allocated":27643904,"data_compressed_original":442220544,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706153,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":149828,"kb_used_data":24444,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94222012,"statfs":{"total":96636764160,"available":96483340288,"internally_reserved":0,"allocated":25030656,"data_stored":392996242,"data_compressed":18653327,"data_compressed_allocated":24551424,"data_compressed_original":392822784,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:16:34.383 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:34.498 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:34.516+0000 7f1e32617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:34.502 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:34.521+0000 7f1e32617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:34.678 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:34.703 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:36.704 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:36.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:36.832+0000 7fbe3bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:36.818 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:36.838+0000 7fbe3bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:36.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~253s 2024-01-11T10:16:36.924 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:36.942 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:37.884 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:37.884 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:37.884 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:37.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:38.006 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.018+0000 7fbb162ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.010 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.029+0000 7fbb162ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.135 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:38.136 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:38.138 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:38.138 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:38.253 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.267+0000 7f2d2ceeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.256 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.275+0000 7f2d2ceeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.270 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:38.270 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:38.270 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:38.270 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:38.270 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:38.271 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:38.273 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:38.273 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:38.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:38.401 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.415+0000 7f857995e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.404 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.423+0000 7f857995e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.408 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:38.412 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:38.412 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:38.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:38.553 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.567+0000 7fdef511e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:38.575+0000 7fdef511e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:38.606 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:38.606 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:38.606 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:38.606 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:38.606 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:38.607 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:38.609 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:38.609 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:38.943 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:39.078 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:39.096+0000 7f4557c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:39.082 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:39.101+0000 7f4557c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:39.234 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:39.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:41.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:41.371 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:41.389+0000 7f1ee66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:41.375 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:41.394+0000 7f1ee66ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:41.594 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:41.612 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:42.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~259s 2024-01-11T10:16:43.611 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:43.611 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:43.611 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:43.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:43.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:43.731 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:43.749+0000 7f93666f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:43.736 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:43.755+0000 7f93666f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:43.750 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:43.764+0000 7fe161aeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:43.754 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:43.772+0000 7fe161aeb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:43.811 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:43.812 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:43.812 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:43.812 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:43.812 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:43.812 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:43.814 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:43.815 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:43.951 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:43.966+0000 7f31c811e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:43.955 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:43.974+0000 7f31c811e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:43.979 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:43.980 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:43.982 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:43.982 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:43.983 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:43.988 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:44.012 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:44.101 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:44.115+0000 7f9d2559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:44.104 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:44.123+0000 7f9d2559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:44.109 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:44.112 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:44.113 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:44.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:44.245+0000 7f71e486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:44.235 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:44.254+0000 7f71e486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:44.249 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:44.250 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:44.252 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:46.013 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:46.128 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:46.146+0000 7f559d35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:46.132 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:46.151+0000 7f559d35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:46.324 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:46.345 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:47.853 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~264s 2024-01-11T10:16:48.346 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:48.486 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:48.503+0000 7f698559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:48.489 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:48.508+0000 7f698559e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:48.686 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:48.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:49.253 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:49.254 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:49.254 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:49.254 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:49.379 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.394+0000 7f56e9217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.383 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.402+0000 7f56e9217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:49.557 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:49.558 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:49.558 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:49.558 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:49.558 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:49.558 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:49.560 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:49.560 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:49.691 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.705+0000 7f7bc2c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.695 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.714+0000 7f7bc2c60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:49.711 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:49.712 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:49.714 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:49.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:49.840 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.854+0000 7fc890217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.844 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.863+0000 7fc890217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.848 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:49.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:49.851 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:49.851 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:49.966 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.980+0000 7f4c71e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:49.989+0000 7f4c71e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:49.984 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:49.984 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:49.985 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:49.985 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:49.985 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:49.985 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:49.986 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:49.986 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:49.987 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:49.989 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:49.990 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:50.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:50.821 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:50.838+0000 7f5d88c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:50.824 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:50.843+0000 7f5d88c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:50.900 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:50.927 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:52.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:53.041 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:53.058+0000 7fbf8af1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:53.046 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:53.065+0000 7fbf8af1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:53.283 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:53.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:53.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~270s 2024-01-11T10:16:54.991 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:16:54.991 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:16:54.991 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:16:54.991 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:16:55.110 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.124+0000 7f002d59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.133+0000 7f002d59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.259 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:55.260 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:55.260 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:55.261 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:16:55.302 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:16:55.384 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.395+0000 7fc90099e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.388 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.407+0000 7fc90099e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:55.411 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:55.414 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:55.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:16:55.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.457+0000 7f92e66de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.444 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.463+0000 7f92e66de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.544 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.558+0000 7fb915406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.548 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.567+0000 7fb915406640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.552 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:55.556 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:55.556 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:16:55.655 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:55.680 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:55.680 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:55.718 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.732+0000 7f13f671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.722 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:55.741+0000 7f13f671e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:16:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:16:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:16:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:16:55.740 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:16:55.741 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:16:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:16:55.743 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:16:57.681 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:16:57.815 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:57.833+0000 7f0b3b315640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:57.819 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:16:57.838+0000 7f0b3b315640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:16:57.977 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:16:58.016 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:16:58.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:16:58.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~275s 2024-01-11T10:17:00.018 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:00.131 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:00.148+0000 7fc4f5e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:00.134 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:00.153+0000 7fc4f5e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:00.337 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:00.356 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:00.356 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:00.744 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:17:00.744 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:17:00.745 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:17:00.861 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:00.877+0000 7fd02ec24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:00.865 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:00.884+0000 7fd02ec24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:01.067 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:17:01.068 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:17:01.079 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-01-11T10:17:00.207696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":241107512,"num_objects":3673,"num_object_clones":0,"num_object_copies":11019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3233,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":31414,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31358,"ondisk_log_size":31358,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":392780,"kb_used_data":46252,"kb_used_omap":0,"kb_used_meta":346496,"kb_avail":377094580,"statfs":{"total":386547056640,"available":386144849920,"internally_reserved":0,"allocated":47362048,"data_stored":730149966,"data_compressed":34647961,"data_compressed_allocated":45608960,"data_compressed_original":729497600,"omap_allocated":0,"internal_metadata":354811904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-158203904,"num_objects":-2414,"num_object_clones":0,"num_object_copies":-7242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2058,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2414,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003123"},"pg_stats":[{"pgid":"5.f","version":"57'2077","reported_seq":2111,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.979988+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:16:57.979988+0000","last_peered":"2024-01-11T10:16:57.979988+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:16:57.979988+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2077,"log_dups_size":0,"ondisk_log_size":2077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2078,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":236}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"57'1975","reported_seq":2009,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:58.083089+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:16:58.083089+0000","last_peered":"2024-01-11T10:16:58.083089+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:16:58.083089+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1975,"log_dups_size":0,"ondisk_log_size":1975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":228,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1976,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":228}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"57'2069","reported_seq":2104,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:58.253796+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:16:58.253796+0000","last_peered":"2024-01-11T10:16:58.253796+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:16:58.253796+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14745624,"num_objects":226,"num_object_clones":0,"num_object_copies":678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":226,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2070,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":226}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"57'1895","reported_seq":1935,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.014443+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:16:57.014443+0000","last_peered":"2024-01-11T10:16:57.014443+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:16:57.014443+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1895,"log_dups_size":0,"ondisk_log_size":1895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15204352,"num_objects":232,"num_object_clones":0,"num_object_copies":696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":232,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1896,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":232}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"57'1851","reported_seq":1887,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:16:58.110478+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:16:58.110478+0000","last_peered":"2024-01-11T10:16:58.110478+0000","last_clean":"2024-01-11T10:16:58.110478+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:16:58.110478+0000","last_undegraded":"2024-01-11T10:16:58.110478+0000","last_fullsized":"2024-01-11T10:16:58.110478+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1851,"log_dups_size":0,"ondisk_log_size":1851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14942208,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1852,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"57'1950","reported_seq":1990,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:56.930111+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:16:56.930111+0000","last_peered":"2024-01-11T10:16:56.930111+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:16:56.930111+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1950,"log_dups_size":0,"ondisk_log_size":1950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16056320,"num_objects":245,"num_object_clones":0,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":245,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1951,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":245}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"57'2004","reported_seq":2038,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.084795+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:16:57.084795+0000","last_peered":"2024-01-11T10:16:57.084795+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:16:57.084795+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2004,"log_dups_size":0,"ondisk_log_size":2004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15400960,"num_objects":235,"num_object_clones":0,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":235,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2005,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":235}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"57'1868","reported_seq":1902,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:58.270157+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:16:58.270157+0000","last_peered":"2024-01-11T10:16:58.270157+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:16:58.270157+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1868,"log_dups_size":0,"ondisk_log_size":1868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15138816,"num_objects":231,"num_object_clones":0,"num_object_copies":693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1869,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":231}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"57'2064","reported_seq":2098,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:58.253441+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:16:58.253441+0000","last_peered":"2024-01-11T10:16:58.253441+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:16:58.253441+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2064,"log_dups_size":0,"ondisk_log_size":2064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14614528,"num_objects":223,"num_object_clones":0,"num_object_copies":669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":223,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2065,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":223}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":223,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:15.097279+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:15.097279+0000","last_peered":"2024-01-11T10:16:15.097279+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:15.097279+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.344665+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:16.344665+0000","last_peered":"2024-01-11T10:16:16.344665+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:16.344665+0000","last_undegraded":"2024-01-11T10:16:16.344665+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"57'1912","reported_seq":1946,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:58.181260+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:16:58.181260+0000","last_peered":"2024-01-11T10:16:58.181260+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:16:58.181260+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1912,"log_dups_size":0,"ondisk_log_size":1912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":14483456,"num_objects":221,"num_object_clones":0,"num_object_copies":663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":221,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1913,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":221}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.303007+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:16:16.303007+0000","last_peered":"2024-01-11T10:16:16.303007+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:16:16.303007+0000","last_undegraded":"2024-01-11T10:16:16.303007+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"57'1915","reported_seq":1955,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.029106+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:16:57.029106+0000","last_peered":"2024-01-11T10:16:57.029106+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:16:57.029106+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1915,"log_dups_size":0,"ondisk_log_size":1915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15466496,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1916,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":236}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.386001+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:16.386001+0000","last_peered":"2024-01-11T10:16:16.386001+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:16.386001+0000","last_undegraded":"2024-01-11T10:16:16.386001+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"57'1857","reported_seq":1897,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.118067+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:16:57.118067+0000","last_peered":"2024-01-11T10:16:57.118067+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:16:57.118067+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1857,"log_dups_size":0,"ondisk_log_size":1857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16515072,"num_objects":252,"num_object_clones":0,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":252,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1858,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":252}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.292134+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:16:16.292134+0000","last_peered":"2024-01-11T10:16:16.292134+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:16:16.292134+0000","last_undegraded":"2024-01-11T10:16:16.292134+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"57'2069","reported_seq":2109,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.148986+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:16:57.148986+0000","last_peered":"2024-01-11T10:16:57.148986+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:16:57.148986+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2070,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":212}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.093416+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:15.093416+0000","last_peered":"2024-01-11T10:16:15.093416+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:15.093416+0000","last_undegraded":"2024-01-11T10:16:15.093416+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"57'1827","reported_seq":1863,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:16:58.109353+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:16:58.109353+0000","last_peered":"2024-01-11T10:16:58.109353+0000","last_clean":"2024-01-11T10:16:58.109353+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:16:58.109353+0000","last_undegraded":"2024-01-11T10:16:58.109353+0000","last_fullsized":"2024-01-11T10:16:58.109353+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1827,"log_dups_size":0,"ondisk_log_size":1827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1828,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.080452+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:15.080452+0000","last_peered":"2024-01-11T10:16:15.080452+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:15.080452+0000","last_undegraded":"2024-01-11T10:16:15.080452+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"57'1935","reported_seq":1975,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.132773+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:16:57.132773+0000","last_peered":"2024-01-11T10:16:57.132773+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:16:57.132773+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1935,"log_dups_size":0,"ondisk_log_size":1935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":15859712,"num_objects":242,"num_object_clones":0,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":242,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1936,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":242}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.320111+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:16:16.320111+0000","last_peered":"2024-01-11T10:16:16.320111+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:16:16.320111+0000","last_undegraded":"2024-01-11T10:16:16.320111+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"57'1993","reported_seq":2027,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:57.991124+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:16:57.991124+0000","last_peered":"2024-01-11T10:16:57.991124+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:16:57.991124+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1993,"log_dups_size":0,"ondisk_log_size":1993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13893632,"num_objects":212,"num_object_clones":0,"num_object_copies":636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1994,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":212}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.338236+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:16.338236+0000","last_peered":"2024-01-11T10:16:16.338236+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:16.338236+0000","last_undegraded":"2024-01-11T10:16:16.338236+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":240517144,"num_objects":3671,"num_object_clones":0,"num_object_copies":11013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3231,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":31277,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":31261,"ondisk_log_size":31261,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":133484,"kb_used_data":12260,"kb_used_omap":0,"kb_used_meta":121216,"kb_avail":94238356,"statfs":{"total":96636764160,"available":96500076544,"internally_reserved":0,"allocated":12554240,"data_stored":192915402,"data_compressed":9154351,"data_compressed_allocated":12050432,"data_compressed_original":192724992,"omap_allocated":0,"internal_metadata":124125184},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48212,"kb_used_data":11212,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":94323628,"statfs":{"total":96636764160,"available":96587395072,"internally_reserved":0,"allocated":11481088,"data_stored":175744970,"data_compressed":8339007,"data_compressed_allocated":10977280,"data_compressed_original":175554560,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706183,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135308,"kb_used_data":9924,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94236532,"statfs":{"total":96636764160,"available":96498208768,"internally_reserved":0,"allocated":10162176,"data_stored":155100562,"data_compressed":7356768,"data_compressed_allocated":9682944,"data_compressed_original":154927104,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:17:02.357 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:17:02.492 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:02.509+0000 7f7bdca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:02.494 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:02.514+0000 7f7bdca6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:02.607 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:02.629 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:04.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~281s 2024-01-11T10:17:04.631 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:04.763 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:04.780+0000 7fa46482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:04.767 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:04.787+0000 7fa46482d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:04.886 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:04.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:06.082 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:17:06.082 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:17:06.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:17:06.199 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:06.215+0000 7f58d862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:06.203 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:06.222+0000 7f58d862d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:06.416 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:17:06.416 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:17:06.428 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2024-01-11T10:17:06.209043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179372600,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2401,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":32356,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32300,"ondisk_log_size":32300,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":393616,"kb_used_data":38768,"kb_used_omap":0,"kb_used_meta":354816,"kb_avail":377093744,"statfs":{"total":386547056640,"available":386143993856,"internally_reserved":0,"allocated":39698432,"data_stored":607532110,"data_compressed":28825409,"data_compressed_allocated":37945344,"data_compressed_original":606879744,"omap_allocated":0,"internal_metadata":363331584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-122880000,"num_objects":-1875,"num_object_clones":0,"num_object_copies":-5625,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1634,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1875,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002869"},"pg_stats":[{"pgid":"5.f","version":"57'2136","reported_seq":2170,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.125462+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:17:03.125462+0000","last_peered":"2024-01-11T10:17:03.125462+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:17:03.125462+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2137,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"57'2026","reported_seq":2060,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.199747+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:17:03.199747+0000","last_peered":"2024-01-11T10:17:03.199747+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:17:03.199747+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2026,"log_dups_size":0,"ondisk_log_size":2026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599872,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":177,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2027,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":177}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"57'2129","reported_seq":2164,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.066381+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:17:03.066381+0000","last_peered":"2024-01-11T10:17:03.066381+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:17:03.066381+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10813464,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2130,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":166}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"57'1949","reported_seq":1989,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:01.830886+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:17:01.830886+0000","last_peered":"2024-01-11T10:17:01.830886+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:17:01.830886+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1949,"log_dups_size":0,"ondisk_log_size":1949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1950,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"57'1909","reported_seq":1945,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:17:03.052347+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:17:03.052347+0000","last_peered":"2024-01-11T10:17:03.052347+0000","last_clean":"2024-01-11T10:17:03.052347+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:17:03.052347+0000","last_undegraded":"2024-01-11T10:17:03.052347+0000","last_fullsized":"2024-01-11T10:17:03.052347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1909,"log_dups_size":0,"ondisk_log_size":1909,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11141120,"num_objects":170,"num_object_clones":0,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1910,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"57'2017","reported_seq":2057,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:01.960807+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:17:01.960807+0000","last_peered":"2024-01-11T10:17:01.960807+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:17:01.960807+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2017,"log_dups_size":0,"ondisk_log_size":2017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11665408,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":178,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":178}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"57'2066","reported_seq":2100,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:01.962613+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:17:01.962613+0000","last_peered":"2024-01-11T10:17:01.962613+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:17:01.962613+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2066,"log_dups_size":0,"ondisk_log_size":2066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2067,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":173}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"57'1919","reported_seq":1953,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.242977+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:17:03.242977+0000","last_peered":"2024-01-11T10:17:03.242977+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:17:03.242977+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1919,"log_dups_size":0,"ondisk_log_size":1919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11796480,"num_objects":180,"num_object_clones":0,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":180,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1920,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":180}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"57'2134","reported_seq":2168,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.200527+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:17:03.200527+0000","last_peered":"2024-01-11T10:17:03.200527+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:17:03.200527+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2134,"log_dups_size":0,"ondisk_log_size":2134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":153,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2135,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":153}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":223,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:15.097279+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:15.097279+0000","last_peered":"2024-01-11T10:16:15.097279+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:15.097279+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.344665+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:16.344665+0000","last_peered":"2024-01-11T10:16:16.344665+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:16.344665+0000","last_undegraded":"2024-01-11T10:16:16.344665+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"57'1972","reported_seq":2006,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.158829+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:17:03.158829+0000","last_peered":"2024-01-11T10:17:03.158829+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:17:03.158829+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1972,"log_dups_size":0,"ondisk_log_size":1972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1973,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":161}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.303007+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:16:16.303007+0000","last_peered":"2024-01-11T10:16:16.303007+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:16:16.303007+0000","last_undegraded":"2024-01-11T10:16:16.303007+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"57'1978","reported_seq":2018,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:02.104873+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:17:02.104873+0000","last_peered":"2024-01-11T10:17:02.104873+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:17:02.104873+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1978,"log_dups_size":0,"ondisk_log_size":1978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11337728,"num_objects":173,"num_object_clones":0,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1979,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":173}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.386001+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:16.386001+0000","last_peered":"2024-01-11T10:16:16.386001+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:16.386001+0000","last_undegraded":"2024-01-11T10:16:16.386001+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"57'1930","reported_seq":1970,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:02.034664+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:17:02.034664+0000","last_peered":"2024-01-11T10:17:02.034664+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:17:02.034664+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1930,"log_dups_size":0,"ondisk_log_size":1930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11730944,"num_objects":179,"num_object_clones":0,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1931,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":179}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.292134+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:16:16.292134+0000","last_peered":"2024-01-11T10:16:16.292134+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:16:16.292134+0000","last_undegraded":"2024-01-11T10:16:16.292134+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"57'2115","reported_seq":2155,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:01.985405+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:17:01.985405+0000","last_peered":"2024-01-11T10:17:01.985405+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:17:01.985405+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2115,"log_dups_size":0,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10878976,"num_objects":166,"num_object_clones":0,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2116,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":166}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.093416+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:15.093416+0000","last_peered":"2024-01-11T10:16:15.093416+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:15.093416+0000","last_undegraded":"2024-01-11T10:16:15.093416+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"57'1879","reported_seq":1915,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:17:03.259546+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:17:03.259546+0000","last_peered":"2024-01-11T10:17:03.259546+0000","last_clean":"2024-01-11T10:17:03.259546+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:17:03.259546+0000","last_undegraded":"2024-01-11T10:17:03.259546+0000","last_fullsized":"2024-01-11T10:17:03.259546+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1879,"log_dups_size":0,"ondisk_log_size":1879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10485760,"num_objects":160,"num_object_clones":0,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1880,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.080452+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:15.080452+0000","last_peered":"2024-01-11T10:16:15.080452+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:15.080452+0000","last_undegraded":"2024-01-11T10:16:15.080452+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"57'1993","reported_seq":2033,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:01.969455+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:17:01.969455+0000","last_peered":"2024-01-11T10:17:01.969455+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:17:01.969455+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1993,"log_dups_size":0,"ondisk_log_size":1993,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12058624,"num_objects":184,"num_object_clones":0,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1994,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":184}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.320111+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:16:16.320111+0000","last_peered":"2024-01-11T10:16:16.320111+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:16:16.320111+0000","last_undegraded":"2024-01-11T10:16:16.320111+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"57'2051","reported_seq":2085,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:03.136878+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:17:03.136878+0000","last_peered":"2024-01-11T10:17:03.136878+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:17:03.136878+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2051,"log_dups_size":0,"ondisk_log_size":2051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10092544,"num_objects":154,"num_object_clones":0,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":154,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2052,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":154}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.338236+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:16.338236+0000","last_peered":"2024-01-11T10:16:16.338236+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:16.338236+0000","last_undegraded":"2024-01-11T10:16:16.338236+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":178782232,"num_objects":2729,"num_object_clones":0,"num_object_copies":8187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2399,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":32219,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32203,"ondisk_log_size":32203,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":135128,"kb_used_data":9744,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94236712,"statfs":{"total":96636764160,"available":96498393088,"internally_reserved":0,"allocated":9977856,"data_stored":151693258,"data_compressed":7196903,"data_compressed_allocated":9474048,"data_compressed_original":151502848,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45604,"kb_used_data":8604,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":94326236,"statfs":{"total":96636764160,"available":96590065664,"internally_reserved":0,"allocated":8810496,"data_stored":133015498,"data_compressed":6309983,"data_compressed_allocated":8306688,"data_compressed_original":132825088,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":137108,"kb_used_data":7564,"kb_used_omap":0,"kb_used_meta":129536,"kb_avail":94234732,"statfs":{"total":96636764160,"available":96496365568,"internally_reserved":0,"allocated":7745536,"data_stored":116434322,"data_compressed":5520688,"data_compressed_allocated":7266304,"data_compressed_original":116260864,"omap_allocated":0,"internal_metadata":132644864},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:17:06.430 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 5.d 5.c 5.a 5.8 5.7 2024-01-11T10:17:06.584 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:06.594+0000 7fc83ce06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:06.588 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:06.607+0000 7fc83ce06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:06.796 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.7,5.8,5.d] on osd.0 to force-backfill; instructing pg(s) [5.a,5.c] on osd.3 to force-backfill; 2024-01-11T10:17:06.796 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:17:06.909 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:17:07.023 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:07.040+0000 7f9806b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:07.026 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:07.046+0000 7f9806b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:07.190 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:07.210 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:09.211 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:09.323 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:09.340+0000 7f42d3cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:09.327 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:09.345+0000 7f42d3cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:09.443 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:09.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:09.871 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~286s 2024-01-11T10:17:11.462 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:17:11.576 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:11.593+0000 7fb3560ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:11.580 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:11.598+0000 7fb3560ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:11.766 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:11.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:11.810 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:17:11.810 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:17:11.811 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:17:11.925 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:11.939+0000 7f9115e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:11.928 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:11.947+0000 7f9115e17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:12.136 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-01-11T10:17:12.137 INFO:teuthology.orchestra.run.smithi040.stderr:dumped all 2024-01-11T10:17:12.149 INFO:teuthology.orchestra.run.smithi040.stdout:{"pg_ready":true,"pg_map":{"version":314,"stamp":"2024-01-11T10:17:10.210339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119013944,"num_objects":1810,"num_object_clones":0,"num_object_copies":5430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1596,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":104,"num_read_kb":92,"num_write":33277,"num_write_kb":1120167,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33221,"ondisk_log_size":33221,"up":52,"acting":52,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":377487360,"kb_used":390000,"kb_used_data":30992,"kb_used_omap":0,"kb_used_meta":358976,"kb_avail":377097360,"statfs":{"total":386547056640,"available":386147696640,"internally_reserved":0,"allocated":31735808,"data_stored":480130126,"data_compressed":22775681,"data_compressed_allocated":29982720,"data_compressed_original":479477760,"omap_allocated":0,"internal_metadata":367591424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-157745152,"num_objects":-2407,"num_object_clones":0,"num_object_copies":-7221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2050,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2407,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003068"},"pg_stats":[{"pgid":"5.f","version":"57'2196","reported_seq":2230,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:07.780376+0000","last_change":"2024-01-11T10:14:18.228338+0000","last_active":"2024-01-11T10:17:07.780376+0000","last_peered":"2024-01-11T10:17:07.780376+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.185857+0000","last_became_peered":"2024-01-11T10:14:17.185857+0000","last_unstale":"2024-01-11T10:17:07.780376+0000","last_undegraded":"2024-01-11T10:14:18.225443+0000","last_fullsized":"2024-01-11T10:14:17.085226+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7667712,"num_objects":117,"num_object_clones":0,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2197,"num_write_kb":74048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":117}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"57'2087","reported_seq":2121,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:08.187523+0000","last_change":"2024-01-11T10:14:18.481757+0000","last_active":"2024-01-11T10:17:08.187523+0000","last_peered":"2024-01-11T10:17:08.187523+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.162129+0000","last_became_peered":"2024-01-11T10:14:17.162129+0000","last_unstale":"2024-01-11T10:17:08.187523+0000","last_undegraded":"2024-01-11T10:14:18.265940+0000","last_fullsized":"2024-01-11T10:14:17.092380+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2087,"log_dups_size":0,"ondisk_log_size":2087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7602176,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2088,"num_write_kb":70528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":116}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"57'2179","reported_seq":2214,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:08.167461+0000","last_change":"2024-01-11T10:14:18.363844+0000","last_active":"2024-01-11T10:17:08.167461+0000","last_peered":"2024-01-11T10:17:08.167461+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.181728+0000","last_became_peered":"2024-01-11T10:14:17.181728+0000","last_unstale":"2024-01-11T10:17:08.167461+0000","last_undegraded":"2024-01-11T10:14:18.224006+0000","last_fullsized":"2024-01-11T10:14:17.082837+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536664,"num_objects":116,"num_object_clones":0,"num_object_copies":348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2180,"num_write_kb":73409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[{"shards":"0,3","objects":116}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.c","version":"57'1999","reported_seq":2039,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:07.115431+0000","last_change":"2024-01-11T10:14:18.353669+0000","last_active":"2024-01-11T10:17:07.115431+0000","last_peered":"2024-01-11T10:17:07.115431+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.100856+0000","last_became_peered":"2024-01-11T10:14:17.100856+0000","last_unstale":"2024-01-11T10:17:07.115431+0000","last_undegraded":"2024-01-11T10:14:18.181402+0000","last_fullsized":"2024-01-11T10:14:17.024510+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1999,"log_dups_size":0,"ondisk_log_size":1999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":128,"num_object_clones":0,"num_object_copies":384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2000,"num_write_kb":68096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":128}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"57'1971","reported_seq":2007,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:17:08.151738+0000","last_change":"2024-01-11T10:14:17.231798+0000","last_active":"2024-01-11T10:17:08.151738+0000","last_peered":"2024-01-11T10:17:08.151738+0000","last_clean":"2024-01-11T10:17:08.151738+0000","last_became_active":"2024-01-11T10:14:17.223312+0000","last_became_peered":"2024-01-11T10:14:17.223312+0000","last_unstale":"2024-01-11T10:17:08.151738+0000","last_undegraded":"2024-01-11T10:17:08.151738+0000","last_fullsized":"2024-01-11T10:17:08.151738+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1971,"log_dups_size":0,"ondisk_log_size":1971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7077888,"num_objects":108,"num_object_clones":0,"num_object_copies":324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1972,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"57'2080","reported_seq":2120,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:06.969529+0000","last_change":"2024-01-11T10:14:18.377613+0000","last_active":"2024-01-11T10:17:06.969529+0000","last_peered":"2024-01-11T10:17:06.969529+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.186612+0000","last_became_peered":"2024-01-11T10:14:17.186612+0000","last_unstale":"2024-01-11T10:17:06.969529+0000","last_undegraded":"2024-01-11T10:14:18.179347+0000","last_fullsized":"2024-01-11T10:14:17.056345+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2080,"log_dups_size":0,"ondisk_log_size":2080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7536640,"num_objects":115,"num_object_clones":0,"num_object_copies":345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2081,"num_write_kb":70272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":115}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"57'2116","reported_seq":2150,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:07.126894+0000","last_change":"2024-01-11T10:14:18.349676+0000","last_active":"2024-01-11T10:17:07.126894+0000","last_peered":"2024-01-11T10:17:07.126894+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.214727+0000","last_became_peered":"2024-01-11T10:14:17.214727+0000","last_unstale":"2024-01-11T10:17:07.126894+0000","last_undegraded":"2024-01-11T10:14:18.149947+0000","last_fullsized":"2024-01-11T10:14:17.063223+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2116,"log_dups_size":0,"ondisk_log_size":2116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2117,"num_write_kb":71680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":123}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"57'1986","reported_seq":2020,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:08.267996+0000","last_change":"2024-01-11T10:14:18.569225+0000","last_active":"2024-01-11T10:17:08.267996+0000","last_peered":"2024-01-11T10:17:08.267996+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.232173+0000","last_became_peered":"2024-01-11T10:14:17.232173+0000","last_unstale":"2024-01-11T10:17:08.267996+0000","last_undegraded":"2024-01-11T10:14:18.546080+0000","last_fullsized":"2024-01-11T10:14:17.089697+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1986,"log_dups_size":0,"ondisk_log_size":1986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7405568,"num_objects":113,"num_object_clones":0,"num_object_copies":339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1987,"num_write_kb":67200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":113}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"57'2193","reported_seq":2227,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:08.058583+0000","last_change":"2024-01-11T10:14:18.289828+0000","last_active":"2024-01-11T10:17:08.058583+0000","last_peered":"2024-01-11T10:17:08.058583+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.188353+0000","last_became_peered":"2024-01-11T10:14:17.188353+0000","last_unstale":"2024-01-11T10:17:08.058583+0000","last_undegraded":"2024-01-11T10:14:18.272563+0000","last_fullsized":"2024-01-11T10:14:17.044461+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6160384,"num_objects":94,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":94,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2194,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":94}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":223,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:16:15.097279+0000","last_change":"2024-01-11T10:12:48.133763+0000","last_active":"2024-01-11T10:16:15.097279+0000","last_peered":"2024-01-11T10:16:15.097279+0000","last_clean":"2024-01-11T10:11:57.077128+0000","last_became_active":"2024-01-11T10:12:48.133763+0000","last_became_peered":"2024-01-11T10:12:48.133763+0000","last_unstale":"2024-01-11T10:16:15.097279+0000","last_undegraded":"2024-01-11T10:12:48.047650+0000","last_fullsized":"2024-01-11T10:12:48.047028+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:19.109457+0000","last_clean_scrub_stamp":"2024-01-11T10:08:19.109457+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.344665+0000","last_change":"2024-01-11T10:12:48.196067+0000","last_active":"2024-01-11T10:16:16.344665+0000","last_peered":"2024-01-11T10:16:16.344665+0000","last_clean":"2024-01-11T10:11:58.304116+0000","last_became_active":"2024-01-11T10:12:48.196067+0000","last_became_peered":"2024-01-11T10:12:48.196067+0000","last_unstale":"2024-01-11T10:16:16.344665+0000","last_undegraded":"2024-01-11T10:16:16.344665+0000","last_fullsized":"2024-01-11T10:12:48.083664+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"57'2032","reported_seq":2066,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:08.256461+0000","last_change":"2024-01-11T10:14:18.480452+0000","last_active":"2024-01-11T10:17:08.256461+0000","last_peered":"2024-01-11T10:17:08.256461+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.276409+0000","last_became_peered":"2024-01-11T10:14:17.276409+0000","last_unstale":"2024-01-11T10:17:08.256461+0000","last_undegraded":"2024-01-11T10:14:18.420942+0000","last_fullsized":"2024-01-11T10:14:17.077482+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2032,"log_dups_size":0,"ondisk_log_size":2032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6619136,"num_objects":101,"num_object_clones":0,"num_object_copies":303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2033,"num_write_kb":68288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":101}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.303007+0000","last_change":"2024-01-11T10:12:48.157741+0000","last_active":"2024-01-11T10:16:16.303007+0000","last_peered":"2024-01-11T10:16:16.303007+0000","last_clean":"2024-01-11T10:11:58.211972+0000","last_became_active":"2024-01-11T10:12:48.157741+0000","last_became_peered":"2024-01-11T10:12:48.157741+0000","last_unstale":"2024-01-11T10:16:16.303007+0000","last_undegraded":"2024-01-11T10:16:16.303007+0000","last_fullsized":"2024-01-11T10:12:48.041220+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"57'2024","reported_seq":2064,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:06.874239+0000","last_change":"2024-01-11T10:14:18.267419+0000","last_active":"2024-01-11T10:17:06.874239+0000","last_peered":"2024-01-11T10:17:06.874239+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.190375+0000","last_became_peered":"2024-01-11T10:14:17.190375+0000","last_unstale":"2024-01-11T10:17:06.874239+0000","last_undegraded":"2024-01-11T10:14:18.179344+0000","last_fullsized":"2024-01-11T10:14:17.029894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2024,"log_dups_size":0,"ondisk_log_size":2024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8323072,"num_objects":127,"num_object_clones":0,"num_object_copies":381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":68864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":127}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.386001+0000","last_change":"2024-01-11T10:12:48.142319+0000","last_active":"2024-01-11T10:16:16.386001+0000","last_peered":"2024-01-11T10:16:16.386001+0000","last_clean":"2024-01-11T10:11:58.293562+0000","last_became_active":"2024-01-11T10:12:48.142319+0000","last_became_peered":"2024-01-11T10:12:48.142319+0000","last_unstale":"2024-01-11T10:16:16.386001+0000","last_undegraded":"2024-01-11T10:16:16.386001+0000","last_fullsized":"2024-01-11T10:12:48.061979+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"57'1998","reported_seq":2038,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:07.106312+0000","last_change":"2024-01-11T10:14:18.272056+0000","last_active":"2024-01-11T10:17:07.106312+0000","last_peered":"2024-01-11T10:17:07.106312+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.105265+0000","last_became_peered":"2024-01-11T10:14:17.105265+0000","last_unstale":"2024-01-11T10:17:07.106312+0000","last_undegraded":"2024-01-11T10:14:18.180813+0000","last_fullsized":"2024-01-11T10:14:17.046265+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1998,"log_dups_size":0,"ondisk_log_size":1998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7274496,"num_objects":111,"num_object_clones":0,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1999,"num_write_kb":67520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":111}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":76,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.292134+0000","last_change":"2024-01-11T10:12:48.174484+0000","last_active":"2024-01-11T10:16:16.292134+0000","last_peered":"2024-01-11T10:16:16.292134+0000","last_clean":"2024-01-11T10:11:58.206011+0000","last_became_active":"2024-01-11T10:12:48.174484+0000","last_became_peered":"2024-01-11T10:12:48.174484+0000","last_unstale":"2024-01-11T10:16:16.292134+0000","last_undegraded":"2024-01-11T10:16:16.292134+0000","last_fullsized":"2024-01-11T10:12:48.060204+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"57'2169","reported_seq":2209,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:07.092113+0000","last_change":"2024-01-11T10:14:18.361188+0000","last_active":"2024-01-11T10:17:07.092113+0000","last_peered":"2024-01-11T10:17:07.092113+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.126610+0000","last_became_peered":"2024-01-11T10:14:17.126610+0000","last_unstale":"2024-01-11T10:17:07.092113+0000","last_undegraded":"2024-01-11T10:14:18.353322+0000","last_fullsized":"2024-01-11T10:14:17.051196+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2169,"log_dups_size":0,"ondisk_log_size":2169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7340032,"num_objects":112,"num_object_clones":0,"num_object_copies":336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2170,"num_write_kb":73024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":112}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":55,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.093416+0000","last_change":"2024-01-11T10:12:48.118981+0000","last_active":"2024-01-11T10:16:15.093416+0000","last_peered":"2024-01-11T10:16:15.093416+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.118981+0000","last_became_peered":"2024-01-11T10:12:48.118981+0000","last_unstale":"2024-01-11T10:16:15.093416+0000","last_undegraded":"2024-01-11T10:16:15.093416+0000","last_fullsized":"2024-01-11T10:12:48.036818+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2","version":"57'1933","reported_seq":1969,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-01-11T10:17:08.256694+0000","last_change":"2024-01-11T10:14:17.211117+0000","last_active":"2024-01-11T10:17:08.256694+0000","last_peered":"2024-01-11T10:17:08.256694+0000","last_clean":"2024-01-11T10:17:08.256694+0000","last_became_active":"2024-01-11T10:14:17.208237+0000","last_became_peered":"2024-01-11T10:14:17.208237+0000","last_unstale":"2024-01-11T10:17:08.256694+0000","last_undegraded":"2024-01-11T10:17:08.256694+0000","last_fullsized":"2024-01-11T10:17:08.256694+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":1933,"log_dups_size":0,"ondisk_log_size":1933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6946816,"num_objects":106,"num_object_clones":0,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1934,"num_write_kb":65280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:15.080452+0000","last_change":"2024-01-11T10:12:48.117726+0000","last_active":"2024-01-11T10:16:15.080452+0000","last_peered":"2024-01-11T10:16:15.080452+0000","last_clean":"2024-01-11T10:11:57.077768+0000","last_became_active":"2024-01-11T10:12:48.117726+0000","last_became_peered":"2024-01-11T10:12:48.117726+0000","last_unstale":"2024-01-11T10:16:15.080452+0000","last_undegraded":"2024-01-11T10:16:15.080452+0000","last_fullsized":"2024-01-11T10:12:48.039399+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"57'2054","reported_seq":2094,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:06.822733+0000","last_change":"2024-01-11T10:14:18.284619+0000","last_active":"2024-01-11T10:17:06.822733+0000","last_peered":"2024-01-11T10:17:06.822733+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.173969+0000","last_became_peered":"2024-01-11T10:14:17.173969+0000","last_unstale":"2024-01-11T10:17:06.822733+0000","last_undegraded":"2024-01-11T10:14:18.181212+0000","last_fullsized":"2024-01-11T10:14:17.033099+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2054,"log_dups_size":0,"ondisk_log_size":2054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8060928,"num_objects":123,"num_object_clones":0,"num_object_copies":369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2055,"num_write_kb":69696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":123}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.320111+0000","last_change":"2024-01-11T10:12:48.161031+0000","last_active":"2024-01-11T10:16:16.320111+0000","last_peered":"2024-01-11T10:16:16.320111+0000","last_clean":"2024-01-11T10:08:21.115873+0000","last_became_active":"2024-01-11T10:12:48.161031+0000","last_became_peered":"2024-01-11T10:12:48.161031+0000","last_unstale":"2024-01-11T10:16:16.320111+0000","last_undegraded":"2024-01-11T10:16:16.320111+0000","last_fullsized":"2024-01-11T10:12:48.074267+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"57'2107","reported_seq":2141,"reported_epoch":57,"state":"active+undersized+degraded","last_fresh":"2024-01-11T10:17:08.127165+0000","last_change":"2024-01-11T10:14:18.425974+0000","last_active":"2024-01-11T10:17:08.127165+0000","last_peered":"2024-01-11T10:17:08.127165+0000","last_clean":"2024-01-11T10:14:15.999573+0000","last_became_active":"2024-01-11T10:14:17.199177+0000","last_became_peered":"2024-01-11T10:14:17.199177+0000","last_unstale":"2024-01-11T10:17:08.127165+0000","last_undegraded":"2024-01-11T10:14:18.230376+0000","last_fullsized":"2024-01-11T10:14:17.040239+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:14:15.999573+0000","last_clean_scrub_stamp":"2024-01-11T10:14:15.999573+0000","objects_scrubbed":0,"log_size":2107,"log_dups_size":0,"ondisk_log_size":2107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6422528,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":98,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2108,"num_write_kb":70592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":98}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":78,"reported_epoch":57,"state":"active+undersized","last_fresh":"2024-01-11T10:16:16.338236+0000","last_change":"2024-01-11T10:12:48.160736+0000","last_active":"2024-01-11T10:16:16.338236+0000","last_peered":"2024-01-11T10:16:16.338236+0000","last_clean":"2024-01-11T10:11:58.305690+0000","last_became_active":"2024-01-11T10:12:48.160736+0000","last_became_peered":"2024-01-11T10:12:48.160736+0000","last_unstale":"2024-01-11T10:16:16.338236+0000","last_undegraded":"2024-01-11T10:16:16.338236+0000","last_fullsized":"2024-01-11T10:12:48.050152+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-11T10:08:21.115873+0000","last_clean_scrub_stamp":"2024-01-11T10:08:21.115873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":16,"stat_sum":{"num_bytes":118423576,"num_objects":1808,"num_object_clones":0,"num_object_copies":5424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1594,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":33140,"num_write_kb":1118273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":33124,"ondisk_log_size":33124,"up":34,"acting":34,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705904,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":75776,"kb_used_data":12856,"kb_used_omap":0,"kb_used_meta":62912,"kb_avail":94296064,"statfs":{"total":96636764160,"available":96559169536,"internally_reserved":0,"allocated":13164544,"data_stored":206389032,"data_compressed":9797835,"data_compressed_allocated":12898304,"data_compressed_original":206290944,"omap_allocated":0,"internal_metadata":64421888},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":9,"seq":38654706193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":132160,"kb_used_data":6776,"kb_used_omap":0,"kb_used_meta":125376,"kb_avail":94239680,"statfs":{"total":96636764160,"available":96501432320,"internally_reserved":0,"allocated":6938624,"data_stored":103065546,"data_compressed":4887799,"data_compressed_allocated":6434816,"data_compressed_original":102875136,"omap_allocated":0,"internal_metadata":128385024},"hb_peers":[1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":9,"seq":38654706193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":47280,"kb_used_data":6120,"kb_used_omap":0,"kb_used_meta":41152,"kb_avail":94324560,"statfs":{"total":96636764160,"available":96588349440,"internally_reserved":0,"allocated":6266880,"data_stored":92317642,"data_compressed":4377431,"data_compressed_allocated":5763072,"data_compressed_original":92127232,"omap_allocated":0,"internal_metadata":42139648},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654706193,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":134784,"kb_used_data":5240,"kb_used_omap":0,"kb_used_meta":129536,"kb_avail":94237056,"statfs":{"total":96636764160,"available":96498745344,"internally_reserved":0,"allocated":5365760,"data_stored":78357906,"data_compressed":3712616,"data_compressed_allocated":4886528,"data_compressed_original":78184448,"omap_allocated":0,"internal_metadata":132644864},"hb_peers":[0,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-11T10:17:12.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-recovery 5.d 5.c 5.9 5.4 5.3 2024-01-11T10:17:12.273 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:12.280+0000 7f9adeb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:12.278 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:12.296+0000 7f9adeb9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:12.473 INFO:teuthology.orchestra.run.smithi040.stderr:instructing pg(s) [5.4,5.d] on osd.0 to force-recovery; instructing pg(s) [5.9] on osd.1 to force-recovery; instructing pg(s) [5.3,5.c] on osd.3 to force-recovery; 2024-01-11T10:17:12.473 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T10:17:13.784 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:13.918 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:13.935+0000 7f33ca55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:13.921 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:13.940+0000 7f33ca55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:14.037 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:14.059 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:15.377 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~292s 2024-01-11T10:17:16.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:17:16.183 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:16.200+0000 7fce7182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:16.187 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:16.205+0000 7fce7182d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:16.353 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:16.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:17.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:17:17.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:17:17.488 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:17:17.488 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:17:17.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Removed 17473 objects 2024-01-11T10:17:17.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Clean up completed and total clean up time :88.3872 2024-01-11T10:17:17.599 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:17.613+0000 7fb00d017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:17.601 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:17.620+0000 7fb00d017640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:17:17.615 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:17:17.616 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:17:17.616 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:17:17.616 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:17:17.616 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:17:17.618 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:17:17.618 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:17.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:17:17.766 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:17.780+0000 7f66b6d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:17.769 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:17.787+0000 7f66b6d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:17:17.783 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:17:17.784 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:17:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:17:17.786 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:17.786 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:17:17.895 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:17.909+0000 7fde0135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:17.898 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:17.916+0000 7fde0135e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:17.901 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:17.904 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:17.905 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:17.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:17:18.012 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:18.025+0000 7ffaaaa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:18.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:18.034+0000 7ffaaaa6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:17:18.030 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:17:18.031 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:17:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:17:18.033 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:18.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:18.508 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:18.525+0000 7f5fc9220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:18.510 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:18.529+0000 7f5fc9220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:18.543 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:18.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:20.563 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:17:20.680 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:20.697+0000 7efedc11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:20.684 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:20.702+0000 7efedc11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:20.717 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:20.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:20.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~297s 2024-01-11T10:17:21.096 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_2 2024-01-11T10:17:21.096 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-01-11T10:17:21.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:21.222+0000 7f93ebc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:21.216 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:21.234+0000 7f93ebc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:22.025 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_2' removed 2024-01-11T10:17:22.041 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-01-11T10:17:22.041 INFO:tasks.radosbench:Beginning radosbench... 2024-01-11T10:17:22.042 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_3 2024-01-11T10:17:22.042 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_3 16 2024-01-11T10:17:22.160 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:22.171+0000 7f6a0bd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:22.162 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:22.181+0000 7f6a0bd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:22.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:22.847 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:22.864+0000 7f92f8115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:22.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:22.868+0000 7f92f8115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:22.883 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:22.902 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:23.029 INFO:teuthology.orchestra.run.smithi040.stderr:pool 'unique_pool_3' created 2024-01-11T10:17:23.034 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:17:23.034 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:17:23.035 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:17:23.035 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:17:23.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-01-11T10:17:23.153 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.166+0000 7f2ca9b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.156 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.174+0000 7f2ca9b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:17:23.193 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:17:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:17:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:17:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:17:23.194 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:17:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:17:23.197 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:23.197 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:17:23.227 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.235+0000 7f419c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.231 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.249+0000 7f419c46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.312 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.326+0000 7f0e94a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.315 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.333+0000 7f0e94a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:17:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:17:23.328 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:17:23.329 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:17:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:17:23.333 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:23.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:17:23.480 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.494+0000 7fb65311e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.483 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.501+0000 7fb65311e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.486 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:23.489 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:23.490 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:23.490 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:17:23.597 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.611+0000 7f6da855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:23.619+0000 7f6da855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:23.613 INFO:teuthology.orchestra.run.smithi040.stderr:no valid command found; 10 closest matches: 2024-01-11T10:17:23.613 INFO:teuthology.orchestra.run.smithi040.stderr:config get 2024-01-11T10:17:23.613 INFO:teuthology.orchestra.run.smithi040.stderr:config help 2024-01-11T10:17:23.613 INFO:teuthology.orchestra.run.smithi040.stderr:config set ... 2024-01-11T10:17:23.613 INFO:teuthology.orchestra.run.smithi040.stderr:config show 2024-01-11T10:17:23.613 INFO:teuthology.orchestra.run.smithi040.stderr:deep_scrub 2024-01-11T10:17:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_ops 2024-01-11T10:17:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:dump_historic_slow_ops 2024-01-11T10:17:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:dump_metrics [] 2024-01-11T10:17:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:dump_ops_in_flight 2024-01-11T10:17:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:dump_pgstate_history 2024-01-11T10:17:23.614 INFO:teuthology.orchestra.run.smithi040.stderr:Error EINVAL: invalid command 2024-01-11T10:17:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-11T10:17:23.616 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:17:24.037 INFO:teuthology.orchestra.run.smithi040.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-01-11T10:17:24.903 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:17:25.013 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:25.030+0000 7ff46219e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:25.016 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:25.034+0000 7ff46219e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:25.049 INFO:teuthology.orchestra.run.smithi040.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-01-11T10:17:25.053 DEBUG:teuthology.orchestra.run.smithi040:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-01-11T10:17:25.055 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-01-11T10:17:25.056 INFO:teuthology.orchestra.run:waiting for 3000 2024-01-11T10:17:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-01-11T10:17:25.098 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:17:25.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:hints = 1 2024-01-11T10:17:25.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-01-11T10:17:25.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:Object prefix: benchmark_data_smithi040_52929 2024-01-11T10:17:25.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:17:25.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 0 0 0 0 0 0 - 0 2024-01-11T10:17:26.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1 16 194 178 11.1242 11.125 0.0549828 0.0863699 2024-01-11T10:17:26.385 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~303s 2024-01-11T10:17:26.386 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-01-11T10:17:26.890 INFO:tasks.ceph.osd.0:Sent signal 15 2024-01-11T10:17:26.890 INFO:tasks.ceph.osd.1:Sent signal 15 2024-01-11T10:17:26.890 INFO:tasks.ceph.osd.3:Sent signal 15 2024-01-11T10:17:26.891 INFO:tasks.ceph.mon.a:Sent signal 15 2024-01-11T10:17:26.891 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-01-11T10:17:26.892 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:17:26.910+0000 7f94db7f7640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 28020) UID: 0 2024-01-11T10:17:26.892 INFO:tasks.ceph.mgr.x.smithi134.stderr:2024-01-11T10:17:26.910+0000 7f94db7f7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-01-11T10:17:26.931 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-01-11T10:17:26.949+0000 7f45fc5fc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 28471) UID: 0 2024-01-11T10:17:26.931 INFO:tasks.ceph.mon.a.smithi040.stderr:2024-01-11T10:17:26.949+0000 7f45fc5fc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-01-11T10:17:27.099 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:17:27.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2 16 374 358 11.1867 11.25 0.0751869 0.0803724 2024-01-11T10:17:27.205 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:27.222+0000 7f0a7286d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:28.109 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3 16 374 358 7.45777 0 - 0.0803724 2024-01-11T10:17:28.617 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:17:28.617 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:17:28.617 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-01-11T10:17:28.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-01-11T10:17:28.739 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:17:28.753+0000 7efd81aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:17:29.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 4 16 374 358 5.5933 0 - 0.0803724 2024-01-11T10:17:30.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 5 16 374 358 4.47458 0 - 0.0803724 2024-01-11T10:17:31.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 6 16 374 358 3.72878 0 - 0.0803724 2024-01-11T10:17:32.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 7 16 374 358 3.19608 0 - 0.0803724 2024-01-11T10:17:33.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 8 16 374 358 2.79656 0 - 0.0803724 2024-01-11T10:17:34.110 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 9 16 374 358 2.48582 0 - 0.0803724 2024-01-11T10:17:35.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 10 16 374 358 2.23723 0 - 0.0803724 2024-01-11T10:17:36.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 11 16 374 358 2.03384 0 - 0.0803724 2024-01-11T10:17:37.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 12 16 374 358 1.86435 0 - 0.0803724 2024-01-11T10:17:38.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 13 16 374 358 1.72093 0 - 0.0803724 2024-01-11T10:17:39.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 14 16 374 358 1.59801 0 - 0.0803724 2024-01-11T10:17:40.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 15 16 374 358 1.49148 0 - 0.0803724 2024-01-11T10:17:41.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 16 16 374 358 1.39826 0 - 0.0803724 2024-01-11T10:17:42.111 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 17 16 374 358 1.31601 0 - 0.0803724 2024-01-11T10:17:43.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 18 16 374 358 1.2429 0 - 0.0803724 2024-01-11T10:17:44.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 19 16 374 358 1.17748 0 - 0.0803724 2024-01-11T10:17:45.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:17:45.131853+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:17:45.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:17:45.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 20 16 374 358 1.11861 0 - 0.0803724 2024-01-11T10:17:46.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 21 16 374 358 1.06534 0 - 0.0803724 2024-01-11T10:17:47.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 22 16 374 358 1.01691 0 - 0.0803724 2024-01-11T10:17:48.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 23 16 374 358 0.972699 0 - 0.0803724 2024-01-11T10:17:49.112 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 24 16 374 358 0.932169 0 - 0.0803724 2024-01-11T10:17:50.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 25 16 374 358 0.894883 0 - 0.0803724 2024-01-11T10:17:51.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 26 16 374 358 0.860464 0 - 0.0803724 2024-01-11T10:17:52.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 27 16 374 358 0.828596 0 - 0.0803724 2024-01-11T10:17:53.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 28 16 374 358 0.799002 0 - 0.0803724 2024-01-11T10:17:54.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 29 16 374 358 0.77145 0 - 0.0803724 2024-01-11T10:17:55.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 30 16 374 358 0.745737 0 - 0.0803724 2024-01-11T10:17:56.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 31 16 374 358 0.721681 0 - 0.0803724 2024-01-11T10:17:57.113 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 32 16 374 358 0.699128 0 - 0.0803724 2024-01-11T10:17:58.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 33 16 374 358 0.677942 0 - 0.0803724 2024-01-11T10:17:59.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 34 16 374 358 0.658003 0 - 0.0803724 2024-01-11T10:18:00.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 35 16 374 358 0.639204 0 - 0.0803724 2024-01-11T10:18:01.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 36 16 374 358 0.621448 0 - 0.0803724 2024-01-11T10:18:02.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 37 16 374 358 0.604652 0 - 0.0803724 2024-01-11T10:18:03.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 38 16 374 358 0.58874 0 - 0.0803724 2024-01-11T10:18:04.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 39 16 374 358 0.573644 0 - 0.0803724 2024-01-11T10:18:05.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:18:05.134419+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:18:05.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:18:05.114 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 40 16 374 358 0.559303 0 - 0.0803724 2024-01-11T10:18:06.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 41 16 374 358 0.545662 0 - 0.0803724 2024-01-11T10:18:07.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 42 16 374 358 0.53267 0 - 0.0803724 2024-01-11T10:18:08.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 43 16 374 358 0.520282 0 - 0.0803724 2024-01-11T10:18:09.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 44 16 374 358 0.508457 0 - 0.0803724 2024-01-11T10:18:10.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 45 16 374 358 0.497158 0 - 0.0803724 2024-01-11T10:18:11.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 46 16 374 358 0.486351 0 - 0.0803724 2024-01-11T10:18:12.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 47 16 374 358 0.476003 0 - 0.0803724 2024-01-11T10:18:13.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 48 16 374 358 0.466086 0 - 0.0803724 2024-01-11T10:18:14.115 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 49 16 374 358 0.456573 0 - 0.0803724 2024-01-11T10:18:15.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 50 16 374 358 0.447442 0 - 0.0803724 2024-01-11T10:18:16.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 51 16 374 358 0.438669 0 - 0.0803724 2024-01-11T10:18:17.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 52 16 374 358 0.430233 0 - 0.0803724 2024-01-11T10:18:18.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 53 16 374 358 0.422115 0 - 0.0803724 2024-01-11T10:18:19.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 54 16 374 358 0.414298 0 - 0.0803724 2024-01-11T10:18:20.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 55 16 374 358 0.406766 0 - 0.0803724 2024-01-11T10:18:21.116 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 56 16 374 358 0.399502 0 - 0.0803724 2024-01-11T10:18:22.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 57 16 374 358 0.392493 0 - 0.0803724 2024-01-11T10:18:23.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 58 16 374 358 0.385725 0 - 0.0803724 2024-01-11T10:18:24.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 59 16 374 358 0.379188 0 - 0.0803724 2024-01-11T10:18:25.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:18:25.137179+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:18:25.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:18:25.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 60 16 374 358 0.372868 0 - 0.0803724 2024-01-11T10:18:26.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 61 16 374 358 0.366755 0 - 0.0803724 2024-01-11T10:18:27.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 62 16 374 358 0.360839 0 - 0.0803724 2024-01-11T10:18:28.117 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 63 16 374 358 0.355112 0 - 0.0803724 2024-01-11T10:18:29.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 64 16 374 358 0.349563 0 - 0.0803724 2024-01-11T10:18:30.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 65 16 374 358 0.344185 0 - 0.0803724 2024-01-11T10:18:31.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 66 16 374 358 0.33897 0 - 0.0803724 2024-01-11T10:18:32.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 67 16 374 358 0.333911 0 - 0.0803724 2024-01-11T10:18:33.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 68 16 374 358 0.329 0 - 0.0803724 2024-01-11T10:18:34.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 69 16 374 358 0.324232 0 - 0.0803724 2024-01-11T10:18:35.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 70 16 374 358 0.3196 0 - 0.0803724 2024-01-11T10:18:36.118 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 71 16 374 358 0.315099 0 - 0.0803724 2024-01-11T10:18:37.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 72 16 374 358 0.310722 0 - 0.0803724 2024-01-11T10:18:38.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 73 16 374 358 0.306466 0 - 0.0803724 2024-01-11T10:18:39.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 74 16 374 358 0.302325 0 - 0.0803724 2024-01-11T10:18:40.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 75 16 374 358 0.298293 0 - 0.0803724 2024-01-11T10:18:41.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 76 16 374 358 0.294369 0 - 0.0803724 2024-01-11T10:18:42.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 77 16 374 358 0.290546 0 - 0.0803724 2024-01-11T10:18:43.119 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 78 16 374 358 0.286821 0 - 0.0803724 2024-01-11T10:18:44.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 79 16 374 358 0.28319 0 - 0.0803724 2024-01-11T10:18:45.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:18:45.139974+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:18:45.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:18:45.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 80 16 374 358 0.27965 0 - 0.0803724 2024-01-11T10:18:46.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 81 16 374 358 0.276198 0 - 0.0803724 2024-01-11T10:18:47.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 82 16 374 358 0.272829 0 - 0.0803724 2024-01-11T10:18:48.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 83 16 374 358 0.269542 0 - 0.0803724 2024-01-11T10:18:49.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 84 16 374 358 0.266333 0 - 0.0803724 2024-01-11T10:18:50.120 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 85 16 374 358 0.2632 0 - 0.0803724 2024-01-11T10:18:51.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 86 16 374 358 0.260139 0 - 0.0803724 2024-01-11T10:18:52.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 87 16 374 358 0.257149 0 - 0.0803724 2024-01-11T10:18:53.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 88 16 374 358 0.254227 0 - 0.0803724 2024-01-11T10:18:54.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 89 16 374 358 0.25137 0 - 0.0803724 2024-01-11T10:18:55.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 90 16 374 358 0.248577 0 - 0.0803724 2024-01-11T10:18:56.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 91 16 374 358 0.245846 0 - 0.0803724 2024-01-11T10:18:57.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 92 16 374 358 0.243174 0 - 0.0803724 2024-01-11T10:18:58.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 93 16 374 358 0.240559 0 - 0.0803724 2024-01-11T10:18:59.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 94 16 374 358 0.238 0 - 0.0803724 2024-01-11T10:19:00.121 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 95 16 374 358 0.235495 0 - 0.0803724 2024-01-11T10:19:01.122 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 96 16 374 358 0.233041 0 - 0.0803724 2024-01-11T10:19:02.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 97 16 374 358 0.230639 0 - 0.0803724 2024-01-11T10:19:03.122 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 98 16 374 358 0.228286 0 - 0.0803724 2024-01-11T10:19:04.122 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 99 16 374 358 0.22598 0 - 0.0803724 2024-01-11T10:19:05.122 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:19:05.142754+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:19:05.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:19:05.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 100 16 374 358 0.22372 0 - 0.0803724 2024-01-11T10:19:06.122 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 101 16 374 358 0.221505 0 - 0.0803724 2024-01-11T10:19:07.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 102 16 374 358 0.219333 0 - 0.0803724 2024-01-11T10:19:08.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 103 16 374 358 0.217204 0 - 0.0803724 2024-01-11T10:19:09.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 104 16 374 358 0.215115 0 - 0.0803724 2024-01-11T10:19:10.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 105 16 374 358 0.213067 0 - 0.0803724 2024-01-11T10:19:11.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 106 16 374 358 0.211056 0 - 0.0803724 2024-01-11T10:19:12.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 107 16 374 358 0.209084 0 - 0.0803724 2024-01-11T10:19:13.123 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 108 16 374 358 0.207148 0 - 0.0803724 2024-01-11T10:19:14.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 109 16 374 358 0.205247 0 - 0.0803724 2024-01-11T10:19:15.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 110 16 374 358 0.203382 0 - 0.0803724 2024-01-11T10:19:16.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 111 16 374 358 0.201549 0 - 0.0803724 2024-01-11T10:19:17.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 112 16 374 358 0.19975 0 - 0.0803724 2024-01-11T10:19:18.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 113 16 374 358 0.197982 0 - 0.0803724 2024-01-11T10:19:19.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 114 16 374 358 0.196245 0 - 0.0803724 2024-01-11T10:19:20.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 115 16 374 358 0.194539 0 - 0.0803724 2024-01-11T10:19:21.124 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 116 16 374 358 0.192862 0 - 0.0803724 2024-01-11T10:19:22.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 117 16 374 358 0.191213 0 - 0.0803724 2024-01-11T10:19:23.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 118 16 374 358 0.189593 0 - 0.0803724 2024-01-11T10:19:24.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 119 16 374 358 0.188 0 - 0.0803724 2024-01-11T10:19:25.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:19:25.145626+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:19:25.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:19:25.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 120 16 374 358 0.186433 0 - 0.0803724 2024-01-11T10:19:26.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 121 16 374 358 0.184892 0 - 0.0803724 2024-01-11T10:19:27.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 122 16 374 358 0.183377 0 - 0.0803724 2024-01-11T10:19:27.131 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:19:27.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:19:28.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 123 16 374 358 0.181886 0 - 0.0803724 2024-01-11T10:19:28.654 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:19:28.655 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:19:28.655 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-01-11T10:19:28.776 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:19:28.791+0000 7f4a3055e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:19:29.125 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 124 16 374 358 0.180419 0 - 0.0803724 2024-01-11T10:19:29.132 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:19:29.249 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:19:29.268+0000 7fb176d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:19:30.126 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 125 16 374 358 0.178976 0 - 0.0803724 2024-01-11T10:19:31.126 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 126 16 374 358 0.177555 0 - 0.0803724 2024-01-11T10:19:32.126 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 127 16 374 358 0.176157 0 - 0.0803724 2024-01-11T10:19:33.126 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 128 16 374 358 0.174781 0 - 0.0803724 2024-01-11T10:19:34.126 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 129 16 374 358 0.173426 0 - 0.0803724 2024-01-11T10:19:35.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 130 16 374 358 0.172092 0 - 0.0803724 2024-01-11T10:19:36.126 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 131 16 374 358 0.170778 0 - 0.0803724 2024-01-11T10:19:37.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 132 16 374 358 0.169484 0 - 0.0803724 2024-01-11T10:19:38.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 133 16 374 358 0.16821 0 - 0.0803724 2024-01-11T10:19:39.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 134 16 374 358 0.166955 0 - 0.0803724 2024-01-11T10:19:40.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 135 16 374 358 0.165718 0 - 0.0803724 2024-01-11T10:19:41.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 136 16 374 358 0.1645 0 - 0.0803724 2024-01-11T10:19:42.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 137 16 374 358 0.163299 0 - 0.0803724 2024-01-11T10:19:43.127 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 138 16 374 358 0.162116 0 - 0.0803724 2024-01-11T10:19:44.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 139 16 374 358 0.160949 0 - 0.0803724 2024-01-11T10:19:45.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:19:45.148471+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:19:45.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:19:45.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 140 16 374 358 0.1598 0 - 0.0803724 2024-01-11T10:19:46.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 141 16 374 358 0.158666 0 - 0.0803724 2024-01-11T10:19:47.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 142 16 374 358 0.157549 0 - 0.0803724 2024-01-11T10:19:48.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 143 16 374 358 0.156447 0 - 0.0803724 2024-01-11T10:19:49.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 144 16 374 358 0.155361 0 - 0.0803724 2024-01-11T10:19:50.128 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 145 16 374 358 0.154289 0 - 0.0803724 2024-01-11T10:19:51.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 146 16 374 358 0.153232 0 - 0.0803724 2024-01-11T10:19:52.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 147 16 374 358 0.15219 0 - 0.0803724 2024-01-11T10:19:53.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 148 16 374 358 0.151162 0 - 0.0803724 2024-01-11T10:19:54.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 149 16 374 358 0.150147 0 - 0.0803724 2024-01-11T10:19:55.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 150 16 374 358 0.149146 0 - 0.0803724 2024-01-11T10:19:56.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 151 16 374 358 0.148158 0 - 0.0803724 2024-01-11T10:19:57.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 152 16 374 358 0.147184 0 - 0.0803724 2024-01-11T10:19:58.129 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 153 16 374 358 0.146222 0 - 0.0803724 2024-01-11T10:19:59.130 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 154 16 374 358 0.145272 0 - 0.0803724 2024-01-11T10:20:00.130 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 155 16 374 358 0.144335 0 - 0.0803724 2024-01-11T10:20:01.130 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 156 16 374 358 0.14341 0 - 0.0803724 2024-01-11T10:20:02.130 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 157 16 374 358 0.142496 0 - 0.0803724 2024-01-11T10:20:03.130 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 158 16 374 358 0.141594 0 - 0.0803724 2024-01-11T10:20:04.130 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 159 16 374 358 0.140704 0 - 0.0803724 2024-01-11T10:20:05.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:20:05.151338+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:20:05.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:20:05.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 160 16 374 358 0.139824 0 - 0.0803724 2024-01-11T10:20:06.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 161 16 374 358 0.138956 0 - 0.0803724 2024-01-11T10:20:07.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 162 16 374 358 0.138098 0 - 0.0803724 2024-01-11T10:20:08.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 163 16 374 358 0.137251 0 - 0.0803724 2024-01-11T10:20:09.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 164 16 374 358 0.136414 0 - 0.0803724 2024-01-11T10:20:10.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 165 16 374 358 0.135587 0 - 0.0803724 2024-01-11T10:20:11.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 166 16 374 358 0.134771 0 - 0.0803724 2024-01-11T10:20:12.131 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 167 16 374 358 0.133964 0 - 0.0803724 2024-01-11T10:20:13.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 168 16 374 358 0.133166 0 - 0.0803724 2024-01-11T10:20:14.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 169 16 374 358 0.132378 0 - 0.0803724 2024-01-11T10:20:15.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 170 16 374 358 0.131599 0 - 0.0803724 2024-01-11T10:20:16.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 171 16 374 358 0.13083 0 - 0.0803724 2024-01-11T10:20:17.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 172 16 374 358 0.130069 0 - 0.0803724 2024-01-11T10:20:18.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 173 16 374 358 0.129317 0 - 0.0803724 2024-01-11T10:20:19.132 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 174 16 374 358 0.128574 0 - 0.0803724 2024-01-11T10:20:20.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 175 16 374 358 0.127839 0 - 0.0803724 2024-01-11T10:20:21.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 176 16 374 358 0.127113 0 - 0.0803724 2024-01-11T10:20:22.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 177 16 374 358 0.126395 0 - 0.0803724 2024-01-11T10:20:23.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 178 16 374 358 0.125685 0 - 0.0803724 2024-01-11T10:20:24.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 179 16 374 358 0.124983 0 - 0.0803724 2024-01-11T10:20:25.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:20:25.154156+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:20:25.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:20:25.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 180 16 374 358 0.124288 0 - 0.0803724 2024-01-11T10:20:26.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 181 16 374 358 0.123602 0 - 0.0803724 2024-01-11T10:20:27.133 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 182 16 374 358 0.122923 0 - 0.0803724 2024-01-11T10:20:28.134 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 183 16 374 358 0.122251 0 - 0.0803724 2024-01-11T10:20:29.134 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 184 16 374 358 0.121586 0 - 0.0803724 2024-01-11T10:20:30.134 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 185 16 374 358 0.120929 0 - 0.0803724 2024-01-11T10:20:31.134 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 186 16 374 358 0.120279 0 - 0.0803724 2024-01-11T10:20:32.134 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 187 16 374 358 0.119636 0 - 0.0803724 2024-01-11T10:20:33.134 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 188 16 374 358 0.119 0 - 0.0803724 2024-01-11T10:20:34.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 189 16 374 358 0.11837 0 - 0.0803724 2024-01-11T10:20:35.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 190 16 374 358 0.117747 0 - 0.0803724 2024-01-11T10:20:36.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 191 16 374 358 0.11713 0 - 0.0803724 2024-01-11T10:20:37.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 192 16 374 358 0.11652 0 - 0.0803724 2024-01-11T10:20:38.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 193 16 374 358 0.115917 0 - 0.0803724 2024-01-11T10:20:39.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 194 16 374 358 0.115319 0 - 0.0803724 2024-01-11T10:20:40.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 195 16 374 358 0.114728 0 - 0.0803724 2024-01-11T10:20:41.135 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 196 16 374 358 0.114142 0 - 0.0803724 2024-01-11T10:20:42.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 197 16 374 358 0.113563 0 - 0.0803724 2024-01-11T10:20:43.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 198 16 374 358 0.112989 0 - 0.0803724 2024-01-11T10:20:44.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 199 16 374 358 0.112422 0 - 0.0803724 2024-01-11T10:20:45.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:20:45.157080+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:20:45.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:20:45.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 200 16 374 358 0.111859 0 - 0.0803724 2024-01-11T10:20:46.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 201 16 374 358 0.111303 0 - 0.0803724 2024-01-11T10:20:47.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 202 16 374 358 0.110752 0 - 0.0803724 2024-01-11T10:20:48.136 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 203 16 374 358 0.110206 0 - 0.0803724 2024-01-11T10:20:49.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 204 16 374 358 0.109666 0 - 0.0803724 2024-01-11T10:20:50.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 205 16 374 358 0.109131 0 - 0.0803724 2024-01-11T10:20:51.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 206 16 374 358 0.108601 0 - 0.0803724 2024-01-11T10:20:52.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 207 16 374 358 0.108077 0 - 0.0803724 2024-01-11T10:20:53.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 208 16 374 358 0.107557 0 - 0.0803724 2024-01-11T10:20:54.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 209 16 374 358 0.107043 0 - 0.0803724 2024-01-11T10:20:55.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 210 16 374 358 0.106533 0 - 0.0803724 2024-01-11T10:20:56.137 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 211 16 374 358 0.106028 0 - 0.0803724 2024-01-11T10:20:57.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 212 16 374 358 0.105528 0 - 0.0803724 2024-01-11T10:20:58.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 213 16 374 358 0.105032 0 - 0.0803724 2024-01-11T10:20:59.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 214 16 374 358 0.104542 0 - 0.0803724 2024-01-11T10:21:00.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 215 16 374 358 0.104055 0 - 0.0803724 2024-01-11T10:21:01.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 216 16 374 358 0.103574 0 - 0.0803724 2024-01-11T10:21:02.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 217 16 374 358 0.103096 0 - 0.0803724 2024-01-11T10:21:03.138 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 218 16 374 358 0.102623 0 - 0.0803724 2024-01-11T10:21:04.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 219 16 374 358 0.102155 0 - 0.0803724 2024-01-11T10:21:05.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:21:05.159908+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:21:05.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:21:05.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 220 16 374 358 0.10169 0 - 0.0803724 2024-01-11T10:21:06.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 221 16 374 358 0.10123 0 - 0.0803724 2024-01-11T10:21:07.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 222 16 374 358 0.100774 0 - 0.0803724 2024-01-11T10:21:08.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 223 16 374 358 0.100322 0 - 0.0803724 2024-01-11T10:21:09.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 224 16 374 358 0.0998745 0 - 0.0803724 2024-01-11T10:21:10.139 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 225 16 374 358 0.0994306 0 - 0.0803724 2024-01-11T10:21:11.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 226 16 374 358 0.0989906 0 - 0.0803724 2024-01-11T10:21:12.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 227 16 374 358 0.0985545 0 - 0.0803724 2024-01-11T10:21:13.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 228 16 374 358 0.0981223 0 - 0.0803724 2024-01-11T10:21:14.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 229 16 374 358 0.0976938 0 - 0.0803724 2024-01-11T10:21:15.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 230 16 374 358 0.097269 0 - 0.0803724 2024-01-11T10:21:16.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 231 16 374 358 0.096848 0 - 0.0803724 2024-01-11T10:21:17.140 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 232 16 374 358 0.0964305 0 - 0.0803724 2024-01-11T10:21:18.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 233 16 374 358 0.0960166 0 - 0.0803724 2024-01-11T10:21:19.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 234 16 374 358 0.0956063 0 - 0.0803724 2024-01-11T10:21:20.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 235 16 374 358 0.0951995 0 - 0.0803724 2024-01-11T10:21:21.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 236 16 374 358 0.0947961 0 - 0.0803724 2024-01-11T10:21:22.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 237 16 374 358 0.0943961 0 - 0.0803724 2024-01-11T10:21:23.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 238 16 374 358 0.0939995 0 - 0.0803724 2024-01-11T10:21:24.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 239 16 374 358 0.0936062 0 - 0.0803724 2024-01-11T10:21:25.141 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:21:25.162767+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:21:25.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:21:25.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 240 16 374 358 0.0932162 0 - 0.0803724 2024-01-11T10:21:26.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 241 16 374 358 0.0928294 0 - 0.0803724 2024-01-11T10:21:27.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 242 16 374 358 0.0924458 0 - 0.0803724 2024-01-11T10:21:28.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 243 16 374 358 0.0920653 0 - 0.0803724 2024-01-11T10:21:28.691 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:21:28.692 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:21:28.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-01-11T10:21:28.814 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:21:28.829+0000 7fea85ede640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:21:29.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 244 16 374 358 0.091688 0 - 0.0803724 2024-01-11T10:21:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:21:29.168 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:21:30.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 245 16 374 358 0.0913138 0 - 0.0803724 2024-01-11T10:21:31.142 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 246 16 374 358 0.0909426 0 - 0.0803724 2024-01-11T10:21:31.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:21:31.283 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:21:31.301+0000 7ff62006d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:21:32.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 247 16 374 358 0.0905744 0 - 0.0803724 2024-01-11T10:21:33.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 248 16 374 358 0.0902092 0 - 0.0803724 2024-01-11T10:21:34.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 249 16 374 358 0.0898469 0 - 0.0803724 2024-01-11T10:21:35.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 250 16 374 358 0.0894875 0 - 0.0803724 2024-01-11T10:21:36.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 251 16 374 358 0.0891309 0 - 0.0803724 2024-01-11T10:21:37.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 252 16 374 358 0.0887772 0 - 0.0803724 2024-01-11T10:21:38.143 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 253 16 374 358 0.0884263 0 - 0.0803724 2024-01-11T10:21:39.144 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 254 16 374 358 0.0880782 0 - 0.0803724 2024-01-11T10:21:40.144 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 255 16 374 358 0.0877328 0 - 0.0803724 2024-01-11T10:21:41.144 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 256 16 374 358 0.0873901 0 - 0.0803724 2024-01-11T10:21:42.144 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 257 16 374 358 0.0870501 0 - 0.0803724 2024-01-11T10:21:43.144 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 258 16 374 358 0.0867126 0 - 0.0803724 2024-01-11T10:21:44.144 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 259 16 374 358 0.0863778 0 - 0.0803724 2024-01-11T10:21:45.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:21:45.165745+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:21:45.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:21:45.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 260 16 374 358 0.0860456 0 - 0.0803724 2024-01-11T10:21:46.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 261 16 374 358 0.0857159 0 - 0.0803724 2024-01-11T10:21:47.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 262 16 374 358 0.0853888 0 - 0.0803724 2024-01-11T10:21:48.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 263 16 374 358 0.0850641 0 - 0.0803724 2024-01-11T10:21:49.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 264 16 374 358 0.0847419 0 - 0.0803724 2024-01-11T10:21:50.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 265 16 374 358 0.0844221 0 - 0.0803724 2024-01-11T10:21:51.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 266 16 374 358 0.0841047 0 - 0.0803724 2024-01-11T10:21:52.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 267 16 374 358 0.0837897 0 - 0.0803724 2024-01-11T10:21:53.145 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 268 16 374 358 0.0834771 0 - 0.0803724 2024-01-11T10:21:54.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 269 16 374 358 0.0831668 0 - 0.0803724 2024-01-11T10:21:55.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 270 16 374 358 0.0828587 0 - 0.0803724 2024-01-11T10:21:56.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 271 16 374 358 0.082553 0 - 0.0803724 2024-01-11T10:21:57.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 272 16 374 358 0.0822495 0 - 0.0803724 2024-01-11T10:21:58.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 273 16 374 358 0.0819482 0 - 0.0803724 2024-01-11T10:21:59.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 274 16 374 358 0.0816491 0 - 0.0803724 2024-01-11T10:22:00.146 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 275 16 374 358 0.0813522 0 - 0.0803724 2024-01-11T10:22:01.147 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 276 16 374 358 0.0810574 0 - 0.0803724 2024-01-11T10:22:02.147 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 277 16 374 358 0.0807648 0 - 0.0803724 2024-01-11T10:22:03.147 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 278 16 374 358 0.0804743 0 - 0.0803724 2024-01-11T10:22:04.147 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 279 16 374 358 0.0801858 0 - 0.0803724 2024-01-11T10:22:05.147 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:22:05.168724+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:22:05.147 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:22:05.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 280 16 374 358 0.0798995 0 - 0.0803724 2024-01-11T10:22:06.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 281 16 374 358 0.0796151 0 - 0.0803724 2024-01-11T10:22:07.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 282 16 374 358 0.0793328 0 - 0.0803724 2024-01-11T10:22:08.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 283 16 374 358 0.0790525 0 - 0.0803724 2024-01-11T10:22:09.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 284 16 374 358 0.0787741 0 - 0.0803724 2024-01-11T10:22:10.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 285 16 374 358 0.0784977 0 - 0.0803724 2024-01-11T10:22:11.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 286 16 374 358 0.0782232 0 - 0.0803724 2024-01-11T10:22:12.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 287 16 374 358 0.0779507 0 - 0.0803724 2024-01-11T10:22:13.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 288 16 374 358 0.07768 0 - 0.0803724 2024-01-11T10:22:14.148 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 289 16 374 358 0.0774112 0 - 0.0803724 2024-01-11T10:22:15.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 290 16 374 358 0.0771443 0 - 0.0803724 2024-01-11T10:22:16.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 291 16 374 358 0.0768792 0 - 0.0803724 2024-01-11T10:22:17.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 292 16 374 358 0.0766159 0 - 0.0803724 2024-01-11T10:22:18.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 293 16 374 358 0.0763544 0 - 0.0803724 2024-01-11T10:22:19.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 294 16 374 358 0.0760947 0 - 0.0803724 2024-01-11T10:22:20.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 295 16 374 358 0.0758368 0 - 0.0803724 2024-01-11T10:22:21.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 296 16 374 358 0.0755806 0 - 0.0803724 2024-01-11T10:22:22.149 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 297 16 374 358 0.0753261 0 - 0.0803724 2024-01-11T10:22:23.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 298 16 374 358 0.0750733 0 - 0.0803724 2024-01-11T10:22:24.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 299 16 374 358 0.0748223 0 - 0.0803724 2024-01-11T10:22:25.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:22:25.171510+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:22:25.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:22:25.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 300 16 374 358 0.0745728 0 - 0.0803724 2024-01-11T10:22:26.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 301 16 374 358 0.0743251 0 - 0.0803724 2024-01-11T10:22:27.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 302 16 374 358 0.074079 0 - 0.0803724 2024-01-11T10:22:28.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 303 16 374 358 0.0738345 0 - 0.0803724 2024-01-11T10:22:29.150 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 304 16 374 358 0.0735916 0 - 0.0803724 2024-01-11T10:22:30.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 305 16 374 358 0.0733503 0 - 0.0803724 2024-01-11T10:22:31.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 306 16 374 358 0.0731106 0 - 0.0803724 2024-01-11T10:22:32.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 307 16 374 358 0.0728725 0 - 0.0803724 2024-01-11T10:22:33.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 308 16 374 358 0.0726359 0 - 0.0803724 2024-01-11T10:22:34.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 309 16 374 358 0.0724008 0 - 0.0803724 2024-01-11T10:22:35.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 310 16 374 358 0.0721673 0 - 0.0803724 2024-01-11T10:22:36.151 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 311 16 374 358 0.0719352 0 - 0.0803724 2024-01-11T10:22:37.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 312 16 374 358 0.0717047 0 - 0.0803724 2024-01-11T10:22:38.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 313 16 374 358 0.0714756 0 - 0.0803724 2024-01-11T10:22:39.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 314 16 374 358 0.0712479 0 - 0.0803724 2024-01-11T10:22:40.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 315 16 374 358 0.0710218 0 - 0.0803724 2024-01-11T10:22:41.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 316 16 374 358 0.070797 0 - 0.0803724 2024-01-11T10:22:42.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 317 16 374 358 0.0705737 0 - 0.0803724 2024-01-11T10:22:43.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 318 16 374 358 0.0703517 0 - 0.0803724 2024-01-11T10:22:44.152 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 319 16 374 358 0.0701312 0 - 0.0803724 2024-01-11T10:22:45.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:22:45.174320+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:22:45.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:22:45.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 320 16 374 358 0.069912 0 - 0.0803724 2024-01-11T10:22:46.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 321 16 374 358 0.0696942 0 - 0.0803724 2024-01-11T10:22:47.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 322 16 374 358 0.0694778 0 - 0.0803724 2024-01-11T10:22:48.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 323 16 374 358 0.0692627 0 - 0.0803724 2024-01-11T10:22:49.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 324 16 374 358 0.0690489 0 - 0.0803724 2024-01-11T10:22:50.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 325 16 374 358 0.0688365 0 - 0.0803724 2024-01-11T10:22:51.153 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 326 16 374 358 0.0686253 0 - 0.0803724 2024-01-11T10:22:52.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 327 16 374 358 0.0684154 0 - 0.0803724 2024-01-11T10:22:53.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 328 16 374 358 0.0682069 0 - 0.0803724 2024-01-11T10:22:54.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 329 16 374 358 0.0679995 0 - 0.0803724 2024-01-11T10:22:55.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 330 16 374 358 0.0677935 0 - 0.0803724 2024-01-11T10:22:56.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 331 16 374 358 0.0675887 0 - 0.0803724 2024-01-11T10:22:57.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 332 16 374 358 0.0673851 0 - 0.0803724 2024-01-11T10:22:58.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 333 16 374 358 0.0671827 0 - 0.0803724 2024-01-11T10:22:59.154 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 334 16 374 358 0.0669816 0 - 0.0803724 2024-01-11T10:23:00.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 335 16 374 358 0.0667816 0 - 0.0803724 2024-01-11T10:23:01.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 336 16 374 358 0.0665829 0 - 0.0803724 2024-01-11T10:23:02.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 337 16 374 358 0.0663853 0 - 0.0803724 2024-01-11T10:23:03.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 338 16 374 358 0.0661889 0 - 0.0803724 2024-01-11T10:23:04.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 339 16 374 358 0.0659937 0 - 0.0803724 2024-01-11T10:23:05.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:23:05.177136+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:23:05.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:23:05.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 340 16 374 358 0.0657996 0 - 0.0803724 2024-01-11T10:23:06.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 341 16 374 358 0.0656066 0 - 0.0803724 2024-01-11T10:23:07.155 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 342 16 374 358 0.0654148 0 - 0.0803724 2024-01-11T10:23:08.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 343 16 374 358 0.0652241 0 - 0.0803724 2024-01-11T10:23:09.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 344 16 374 358 0.0650345 0 - 0.0803724 2024-01-11T10:23:10.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 345 16 374 358 0.064846 0 - 0.0803724 2024-01-11T10:23:11.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 346 16 374 358 0.0646586 0 - 0.0803724 2024-01-11T10:23:12.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 347 16 374 358 0.0644722 0 - 0.0803724 2024-01-11T10:23:13.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 348 16 374 358 0.064287 0 - 0.0803724 2024-01-11T10:23:14.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 349 16 374 358 0.0641028 0 - 0.0803724 2024-01-11T10:23:15.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 350 16 374 358 0.0639196 0 - 0.0803724 2024-01-11T10:23:16.156 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 351 16 374 358 0.0637375 0 - 0.0803724 2024-01-11T10:23:17.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 352 16 374 358 0.0635565 0 - 0.0803724 2024-01-11T10:23:18.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 353 16 374 358 0.0633764 0 - 0.0803724 2024-01-11T10:23:19.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 354 16 374 358 0.0631974 0 - 0.0803724 2024-01-11T10:23:20.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 355 16 374 358 0.0630194 0 - 0.0803724 2024-01-11T10:23:21.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 356 16 374 358 0.0628424 0 - 0.0803724 2024-01-11T10:23:22.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 357 16 374 358 0.0626663 0 - 0.0803724 2024-01-11T10:23:23.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 358 16 374 358 0.0624913 0 - 0.0803724 2024-01-11T10:23:24.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 359 16 374 358 0.0623172 0 - 0.0803724 2024-01-11T10:23:25.157 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:23:25.179444+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:23:25.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:23:25.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 360 16 374 358 0.0621441 0 - 0.0803724 2024-01-11T10:23:26.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 361 16 374 358 0.061972 0 - 0.0803724 2024-01-11T10:23:27.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 362 16 374 358 0.0618008 0 - 0.0803724 2024-01-11T10:23:28.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 363 16 374 358 0.0616305 0 - 0.0803724 2024-01-11T10:23:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:23:28.730 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:23:28.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-01-11T10:23:28.850 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:23:28.867+0000 7fe0140ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:23:29.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 364 16 374 358 0.0614612 0 - 0.0803724 2024-01-11T10:23:30.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 365 16 374 358 0.0612929 0 - 0.0803724 2024-01-11T10:23:31.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 366 16 374 358 0.0611254 0 - 0.0803724 2024-01-11T10:23:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:23:31.204 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:23:32.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 367 16 374 358 0.0609588 0 - 0.0803724 2024-01-11T10:23:33.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 368 16 374 358 0.0607932 0 - 0.0803724 2024-01-11T10:23:33.204 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:23:33.322 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:23:33.342+0000 7fabd8cd5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:23:34.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 369 16 374 358 0.0606285 0 - 0.0803724 2024-01-11T10:23:35.158 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 370 16 374 358 0.0604646 0 - 0.0803724 2024-01-11T10:23:36.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 371 16 374 358 0.0603016 0 - 0.0803724 2024-01-11T10:23:37.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 372 16 374 358 0.0601395 0 - 0.0803724 2024-01-11T10:23:38.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 373 16 374 358 0.0599783 0 - 0.0803724 2024-01-11T10:23:39.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 374 16 374 358 0.0598179 0 - 0.0803724 2024-01-11T10:23:40.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 375 16 374 358 0.0596584 0 - 0.0803724 2024-01-11T10:23:41.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 376 16 374 358 0.0594998 0 - 0.0803724 2024-01-11T10:23:42.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 377 16 374 358 0.059342 0 - 0.0803724 2024-01-11T10:23:43.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 378 16 374 358 0.059185 0 - 0.0803724 2024-01-11T10:23:44.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 379 16 374 358 0.0590288 0 - 0.0803724 2024-01-11T10:23:45.159 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:23:45.181550+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:23:45.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:23:45.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 380 16 374 358 0.0588735 0 - 0.0803724 2024-01-11T10:23:46.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 381 16 374 358 0.058719 0 - 0.0803724 2024-01-11T10:23:47.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 382 16 374 358 0.0585653 0 - 0.0803724 2024-01-11T10:23:48.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 383 16 374 358 0.0584123 0 - 0.0803724 2024-01-11T10:23:49.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 384 16 374 358 0.0582602 0 - 0.0803724 2024-01-11T10:23:50.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 385 16 374 358 0.0581089 0 - 0.0803724 2024-01-11T10:23:51.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 386 16 374 358 0.0579584 0 - 0.0803724 2024-01-11T10:23:52.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 387 16 374 358 0.0578086 0 - 0.0803724 2024-01-11T10:23:53.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 388 16 374 358 0.0576596 0 - 0.0803724 2024-01-11T10:23:54.160 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 389 16 374 358 0.0575114 0 - 0.0803724 2024-01-11T10:23:55.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 390 16 374 358 0.057364 0 - 0.0803724 2024-01-11T10:23:56.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 391 16 374 358 0.0572173 0 - 0.0803724 2024-01-11T10:23:57.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 392 16 374 358 0.0570713 0 - 0.0803724 2024-01-11T10:23:58.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 393 16 374 358 0.0569261 0 - 0.0803724 2024-01-11T10:23:59.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 394 16 374 358 0.0567816 0 - 0.0803724 2024-01-11T10:24:00.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 395 16 374 358 0.0566379 0 - 0.0803724 2024-01-11T10:24:01.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 396 16 374 358 0.0564948 0 - 0.0803724 2024-01-11T10:24:02.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 397 16 374 358 0.0563525 0 - 0.0803724 2024-01-11T10:24:03.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 398 16 374 358 0.056211 0 - 0.0803724 2024-01-11T10:24:04.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 399 16 374 358 0.0560701 0 - 0.0803724 2024-01-11T10:24:05.161 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:24:05.183598+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:24:05.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:24:05.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 400 16 374 358 0.0559299 0 - 0.0803724 2024-01-11T10:24:06.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 401 16 374 358 0.0557904 0 - 0.0803724 2024-01-11T10:24:07.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 402 16 374 358 0.0556517 0 - 0.0803724 2024-01-11T10:24:08.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 403 16 374 358 0.0555136 0 - 0.0803724 2024-01-11T10:24:09.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 404 16 374 358 0.0553762 0 - 0.0803724 2024-01-11T10:24:10.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 405 16 374 358 0.0552394 0 - 0.0803724 2024-01-11T10:24:11.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 406 16 374 358 0.0551034 0 - 0.0803724 2024-01-11T10:24:12.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 407 16 374 358 0.054968 0 - 0.0803724 2024-01-11T10:24:13.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 408 16 374 358 0.0548333 0 - 0.0803724 2024-01-11T10:24:14.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 409 16 374 358 0.0546992 0 - 0.0803724 2024-01-11T10:24:15.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 410 16 374 358 0.0545658 0 - 0.0803724 2024-01-11T10:24:16.162 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 411 16 374 358 0.0544331 0 - 0.0803724 2024-01-11T10:24:17.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 412 16 374 358 0.0543009 0 - 0.0803724 2024-01-11T10:24:18.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 413 16 374 358 0.0541695 0 - 0.0803724 2024-01-11T10:24:19.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 414 16 374 358 0.0540386 0 - 0.0803724 2024-01-11T10:24:20.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 415 16 374 358 0.0539084 0 - 0.0803724 2024-01-11T10:24:21.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 416 16 374 358 0.0537788 0 - 0.0803724 2024-01-11T10:24:22.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 417 16 374 358 0.0536499 0 - 0.0803724 2024-01-11T10:24:23.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 418 16 374 358 0.0535215 0 - 0.0803724 2024-01-11T10:24:24.163 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 419 16 374 358 0.0533938 0 - 0.0803724 2024-01-11T10:24:25.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:24:25.185625+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:24:25.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:24:25.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 420 16 374 358 0.0532667 0 - 0.0803724 2024-01-11T10:24:26.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 421 16 374 358 0.0531401 0 - 0.0803724 2024-01-11T10:24:27.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 422 16 374 358 0.0530142 0 - 0.0803724 2024-01-11T10:24:28.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 423 16 374 358 0.0528889 0 - 0.0803724 2024-01-11T10:24:29.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 424 16 374 358 0.0527642 0 - 0.0803724 2024-01-11T10:24:30.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 425 16 374 358 0.05264 0 - 0.0803724 2024-01-11T10:24:31.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 426 16 374 358 0.0525165 0 - 0.0803724 2024-01-11T10:24:32.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 427 16 374 358 0.0523935 0 - 0.0803724 2024-01-11T10:24:33.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 428 16 374 358 0.0522711 0 - 0.0803724 2024-01-11T10:24:34.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 429 16 374 358 0.0521492 0 - 0.0803724 2024-01-11T10:24:35.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 430 16 374 358 0.0520279 0 - 0.0803724 2024-01-11T10:24:36.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 431 16 374 358 0.0519072 0 - 0.0803724 2024-01-11T10:24:37.164 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 432 16 374 358 0.0517871 0 - 0.0803724 2024-01-11T10:24:38.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 433 16 374 358 0.0516675 0 - 0.0803724 2024-01-11T10:24:39.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 434 16 374 358 0.0515484 0 - 0.0803724 2024-01-11T10:24:40.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 435 16 374 358 0.0514299 0 - 0.0803724 2024-01-11T10:24:41.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 436 16 374 358 0.051312 0 - 0.0803724 2024-01-11T10:24:42.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 437 16 374 358 0.0511946 0 - 0.0803724 2024-01-11T10:24:43.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 438 16 374 358 0.0510777 0 - 0.0803724 2024-01-11T10:24:44.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 439 16 374 358 0.0509613 0 - 0.0803724 2024-01-11T10:24:45.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:24:45.187661+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:24:45.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:24:45.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 440 16 374 358 0.0508455 0 - 0.0803724 2024-01-11T10:24:46.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 441 16 374 358 0.0507302 0 - 0.0803724 2024-01-11T10:24:47.165 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 442 16 374 358 0.0506155 0 - 0.0803724 2024-01-11T10:24:48.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 443 16 374 358 0.0505012 0 - 0.0803724 2024-01-11T10:24:49.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 444 16 374 358 0.0503875 0 - 0.0803724 2024-01-11T10:24:50.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 445 16 374 358 0.0502742 0 - 0.0803724 2024-01-11T10:24:51.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 446 16 374 358 0.0501615 0 - 0.0803724 2024-01-11T10:24:52.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 447 16 374 358 0.0500493 0 - 0.0803724 2024-01-11T10:24:53.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 448 16 374 358 0.0499376 0 - 0.0803724 2024-01-11T10:24:54.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 449 16 374 358 0.0498264 0 - 0.0803724 2024-01-11T10:24:55.166 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 450 16 374 358 0.0497157 0 - 0.0803724 2024-01-11T10:24:56.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 451 16 374 358 0.0496054 0 - 0.0803724 2024-01-11T10:24:57.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 452 16 374 358 0.0494957 0 - 0.0803724 2024-01-11T10:24:58.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 453 16 374 358 0.0493864 0 - 0.0803724 2024-01-11T10:24:59.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 454 16 374 358 0.0492776 0 - 0.0803724 2024-01-11T10:25:00.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 455 16 374 358 0.0491693 0 - 0.0803724 2024-01-11T10:25:01.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 456 16 374 358 0.0490615 0 - 0.0803724 2024-01-11T10:25:02.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 457 16 374 358 0.0489542 0 - 0.0803724 2024-01-11T10:25:03.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 458 16 374 358 0.0488473 0 - 0.0803724 2024-01-11T10:25:04.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 459 16 374 358 0.0487409 0 - 0.0803724 2024-01-11T10:25:05.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:25:05.189733+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:25:05.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:25:05.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 460 16 374 358 0.0486349 0 - 0.0803724 2024-01-11T10:25:06.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 461 16 374 358 0.0485294 0 - 0.0803724 2024-01-11T10:25:07.167 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 462 16 374 358 0.0484244 0 - 0.0803724 2024-01-11T10:25:08.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 463 16 374 358 0.0483198 0 - 0.0803724 2024-01-11T10:25:09.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 464 16 374 358 0.0482157 0 - 0.0803724 2024-01-11T10:25:10.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 465 16 374 358 0.048112 0 - 0.0803724 2024-01-11T10:25:11.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 466 16 374 358 0.0480087 0 - 0.0803724 2024-01-11T10:25:12.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 467 16 374 358 0.0479059 0 - 0.0803724 2024-01-11T10:25:13.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 468 16 374 358 0.0478036 0 - 0.0803724 2024-01-11T10:25:14.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 469 16 374 358 0.0477017 0 - 0.0803724 2024-01-11T10:25:15.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 470 16 374 358 0.0476002 0 - 0.0803724 2024-01-11T10:25:16.168 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 471 16 374 358 0.0474991 0 - 0.0803724 2024-01-11T10:25:17.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 472 16 374 358 0.0473985 0 - 0.0803724 2024-01-11T10:25:18.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 473 16 374 358 0.0472983 0 - 0.0803724 2024-01-11T10:25:19.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 474 16 374 358 0.0471985 0 - 0.0803724 2024-01-11T10:25:20.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 475 16 374 358 0.0470991 0 - 0.0803724 2024-01-11T10:25:21.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 476 16 374 358 0.0470002 0 - 0.0803724 2024-01-11T10:25:22.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 477 16 374 358 0.0469016 0 - 0.0803724 2024-01-11T10:25:23.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 478 16 374 358 0.0468035 0 - 0.0803724 2024-01-11T10:25:24.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 479 16 374 358 0.0467058 0 - 0.0803724 2024-01-11T10:25:25.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:25:25.191824+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:25:25.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:25:25.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 480 16 374 358 0.0466085 0 - 0.0803724 2024-01-11T10:25:26.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 481 16 374 358 0.0465116 0 - 0.0803724 2024-01-11T10:25:27.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 482 16 374 358 0.0464151 0 - 0.0803724 2024-01-11T10:25:28.169 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 483 16 374 358 0.046319 0 - 0.0803724 2024-01-11T10:25:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:25:28.767 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-01-11T10:25:29.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 484 16 374 358 0.0462233 0 - 0.0803724 2024-01-11T10:25:30.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 485 16 374 358 0.046128 0 - 0.0803724 2024-01-11T10:25:31.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 486 16 374 358 0.0460331 0 - 0.0803724 2024-01-11T10:25:32.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 487 16 374 358 0.0459386 0 - 0.0803724 2024-01-11T10:25:33.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 488 16 374 358 0.0458445 0 - 0.0803724 2024-01-11T10:25:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:25:33.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:25:33.768 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2024-01-11T10:25:33.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.00 2024-01-11T10:25:33.769 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T10:25:33.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:25:33.907+0000 7f027a55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:25:34.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 489 16 374 358 0.0457507 0 - 0.0803724 2024-01-11T10:25:35.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 490 16 374 358 0.0456573 0 - 0.0803724 2024-01-11T10:25:35.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:25:35.363 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:25:35.382+0000 7fc13f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:25:36.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 491 16 374 358 0.0455644 0 - 0.0803724 2024-01-11T10:25:37.170 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 492 16 374 358 0.0454718 0 - 0.0803724 2024-01-11T10:25:38.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 493 16 374 358 0.0453795 0 - 0.0803724 2024-01-11T10:25:39.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 494 16 374 358 0.0452877 0 - 0.0803724 2024-01-11T10:25:40.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 495 16 374 358 0.0451962 0 - 0.0803724 2024-01-11T10:25:41.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 496 16 374 358 0.0451051 0 - 0.0803724 2024-01-11T10:25:42.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 497 16 374 358 0.0450143 0 - 0.0803724 2024-01-11T10:25:43.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 498 16 374 358 0.0449239 0 - 0.0803724 2024-01-11T10:25:44.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 499 16 374 358 0.0448339 0 - 0.0803724 2024-01-11T10:25:45.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:25:45.193761+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:25:45.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:25:45.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 500 16 374 358 0.0447442 0 - 0.0803724 2024-01-11T10:25:46.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 501 16 374 358 0.0446549 0 - 0.0803724 2024-01-11T10:25:47.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 502 16 374 358 0.044566 0 - 0.0803724 2024-01-11T10:25:48.171 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 503 16 374 358 0.0444774 0 - 0.0803724 2024-01-11T10:25:49.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 504 16 374 358 0.0443891 0 - 0.0803724 2024-01-11T10:25:50.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 505 16 374 358 0.0443012 0 - 0.0803724 2024-01-11T10:25:51.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 506 16 374 358 0.0442137 0 - 0.0803724 2024-01-11T10:25:52.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 507 16 374 358 0.0441265 0 - 0.0803724 2024-01-11T10:25:53.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 508 16 374 358 0.0440396 0 - 0.0803724 2024-01-11T10:25:54.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 509 16 374 358 0.0439531 0 - 0.0803724 2024-01-11T10:25:55.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 510 16 374 358 0.0438669 0 - 0.0803724 2024-01-11T10:25:56.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 511 16 374 358 0.0437811 0 - 0.0803724 2024-01-11T10:25:57.172 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 512 16 374 358 0.0436956 0 - 0.0803724 2024-01-11T10:25:58.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 513 16 374 358 0.0436104 0 - 0.0803724 2024-01-11T10:25:59.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 514 16 374 358 0.0435255 0 - 0.0803724 2024-01-11T10:26:00.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 515 16 374 358 0.043441 0 - 0.0803724 2024-01-11T10:26:01.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 516 16 374 358 0.0433568 0 - 0.0803724 2024-01-11T10:26:02.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 517 16 374 358 0.043273 0 - 0.0803724 2024-01-11T10:26:03.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 518 16 374 358 0.0431895 0 - 0.0803724 2024-01-11T10:26:04.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 519 16 374 358 0.0431062 0 - 0.0803724 2024-01-11T10:26:05.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:26:05.195799+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:26:05.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:26:05.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 520 16 374 358 0.0430233 0 - 0.0803724 2024-01-11T10:26:06.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 521 16 374 358 0.0429408 0 - 0.0803724 2024-01-11T10:26:07.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 522 16 374 358 0.0428585 0 - 0.0803724 2024-01-11T10:26:08.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 523 16 374 358 0.0427766 0 - 0.0803724 2024-01-11T10:26:09.173 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 524 16 374 358 0.0426949 0 - 0.0803724 2024-01-11T10:26:10.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 525 16 374 358 0.0426136 0 - 0.0803724 2024-01-11T10:26:11.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 526 16 374 358 0.0425326 0 - 0.0803724 2024-01-11T10:26:12.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 527 16 374 358 0.0424519 0 - 0.0803724 2024-01-11T10:26:13.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 528 16 374 358 0.0423715 0 - 0.0803724 2024-01-11T10:26:14.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 529 16 374 358 0.0422914 0 - 0.0803724 2024-01-11T10:26:15.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 530 16 374 358 0.0422116 0 - 0.0803724 2024-01-11T10:26:16.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 531 16 374 358 0.0421321 0 - 0.0803724 2024-01-11T10:26:17.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 532 16 374 358 0.0420529 0 - 0.0803724 2024-01-11T10:26:18.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 533 16 374 358 0.041974 0 - 0.0803724 2024-01-11T10:26:19.174 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 534 16 374 358 0.0418954 0 - 0.0803724 2024-01-11T10:26:20.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 535 16 374 358 0.0418171 0 - 0.0803724 2024-01-11T10:26:21.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 536 16 374 358 0.0417391 0 - 0.0803724 2024-01-11T10:26:22.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 537 16 374 358 0.0416614 0 - 0.0803724 2024-01-11T10:26:23.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 538 16 374 358 0.0415839 0 - 0.0803724 2024-01-11T10:26:24.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 539 16 374 358 0.0415068 0 - 0.0803724 2024-01-11T10:26:25.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:26:25.197862+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:26:25.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:26:25.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 540 16 374 358 0.0414299 0 - 0.0803724 2024-01-11T10:26:26.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 541 16 374 358 0.0413533 0 - 0.0803724 2024-01-11T10:26:27.175 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 542 16 374 358 0.0412771 0 - 0.0803724 2024-01-11T10:26:28.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 543 16 374 358 0.041201 0 - 0.0803724 2024-01-11T10:26:29.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 544 16 374 358 0.0411253 0 - 0.0803724 2024-01-11T10:26:30.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 545 16 374 358 0.0410498 0 - 0.0803724 2024-01-11T10:26:31.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 546 16 374 358 0.0409747 0 - 0.0803724 2024-01-11T10:26:32.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 547 16 374 358 0.0408998 0 - 0.0803724 2024-01-11T10:26:33.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 548 16 374 358 0.0408251 0 - 0.0803724 2024-01-11T10:26:34.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 549 16 374 358 0.0407508 0 - 0.0803724 2024-01-11T10:26:35.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 550 16 374 358 0.0406767 0 - 0.0803724 2024-01-11T10:26:36.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 551 16 374 358 0.0406029 0 - 0.0803724 2024-01-11T10:26:37.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 552 16 374 358 0.0405293 0 - 0.0803724 2024-01-11T10:26:38.176 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 553 16 374 358 0.040456 0 - 0.0803724 2024-01-11T10:26:39.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 554 16 374 358 0.040383 0 - 0.0803724 2024-01-11T10:26:40.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 555 16 374 358 0.0403102 0 - 0.0803724 2024-01-11T10:26:41.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 556 16 374 358 0.0402377 0 - 0.0803724 2024-01-11T10:26:42.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 557 16 374 358 0.0401655 0 - 0.0803724 2024-01-11T10:26:43.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 558 16 374 358 0.0400935 0 - 0.0803724 2024-01-11T10:26:44.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 559 16 374 358 0.0400218 0 - 0.0803724 2024-01-11T10:26:45.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:26:45.199877+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:26:45.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:26:45.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 560 16 374 358 0.0399503 0 - 0.0803724 2024-01-11T10:26:46.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 561 16 374 358 0.0398791 0 - 0.0803724 2024-01-11T10:26:47.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 562 16 374 358 0.0398082 0 - 0.0803724 2024-01-11T10:26:48.177 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 563 16 374 358 0.0397374 0 - 0.0803724 2024-01-11T10:26:49.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 564 16 374 358 0.039667 0 - 0.0803724 2024-01-11T10:26:50.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 565 16 374 358 0.0395968 0 - 0.0803724 2024-01-11T10:26:51.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 566 16 374 358 0.0395268 0 - 0.0803724 2024-01-11T10:26:52.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 567 16 374 358 0.0394571 0 - 0.0803724 2024-01-11T10:26:53.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 568 16 374 358 0.0393877 0 - 0.0803724 2024-01-11T10:26:54.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 569 16 374 358 0.0393184 0 - 0.0803724 2024-01-11T10:26:55.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 570 16 374 358 0.0392495 0 - 0.0803724 2024-01-11T10:26:56.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 571 16 374 358 0.0391807 0 - 0.0803724 2024-01-11T10:26:57.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 572 16 374 358 0.0391122 0 - 0.0803724 2024-01-11T10:26:58.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 573 16 374 358 0.039044 0 - 0.0803724 2024-01-11T10:26:59.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 574 16 374 358 0.0389759 0 - 0.0803724 2024-01-11T10:27:00.178 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 575 16 374 358 0.0389082 0 - 0.0803724 2024-01-11T10:27:01.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 576 16 374 358 0.0388406 0 - 0.0803724 2024-01-11T10:27:02.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 577 16 374 358 0.0387733 0 - 0.0803724 2024-01-11T10:27:03.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 578 16 374 358 0.0387062 0 - 0.0803724 2024-01-11T10:27:04.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 579 16 374 358 0.0386394 0 - 0.0803724 2024-01-11T10:27:05.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:27:05.201932+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:27:05.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:27:05.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 580 16 374 358 0.0385728 0 - 0.0803724 2024-01-11T10:27:06.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 581 16 374 358 0.0385064 0 - 0.0803724 2024-01-11T10:27:07.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 582 16 374 358 0.0384402 0 - 0.0803724 2024-01-11T10:27:08.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 583 16 374 358 0.0383743 0 - 0.0803724 2024-01-11T10:27:09.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 584 16 374 358 0.0383086 0 - 0.0803724 2024-01-11T10:27:10.179 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 585 16 374 358 0.0382431 0 - 0.0803724 2024-01-11T10:27:11.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 586 16 374 358 0.0381778 0 - 0.0803724 2024-01-11T10:27:12.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 587 16 374 358 0.0381128 0 - 0.0803724 2024-01-11T10:27:13.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 588 16 374 358 0.038048 0 - 0.0803724 2024-01-11T10:27:14.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 589 16 374 358 0.0379834 0 - 0.0803724 2024-01-11T10:27:15.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 590 16 374 358 0.037919 0 - 0.0803724 2024-01-11T10:27:16.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 591 16 374 358 0.0378548 0 - 0.0803724 2024-01-11T10:27:17.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 592 16 374 358 0.0377909 0 - 0.0803724 2024-01-11T10:27:18.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 593 16 374 358 0.0377272 0 - 0.0803724 2024-01-11T10:27:19.180 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 594 16 374 358 0.0376637 0 - 0.0803724 2024-01-11T10:27:20.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 595 16 374 358 0.0376004 0 - 0.0803724 2024-01-11T10:27:21.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 596 16 374 358 0.0375373 0 - 0.0803724 2024-01-11T10:27:22.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 597 16 374 358 0.0374744 0 - 0.0803724 2024-01-11T10:27:23.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 598 16 374 358 0.0374117 0 - 0.0803724 2024-01-11T10:27:24.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 599 16 374 358 0.0373493 0 - 0.0803724 2024-01-11T10:27:25.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:27:25.203984+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:27:25.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:27:25.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 600 16 374 358 0.037287 0 - 0.0803724 2024-01-11T10:27:26.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 601 16 374 358 0.037225 0 - 0.0803724 2024-01-11T10:27:27.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 602 16 374 358 0.0371632 0 - 0.0803724 2024-01-11T10:27:28.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 603 16 374 358 0.0371015 0 - 0.0803724 2024-01-11T10:27:29.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 604 16 374 358 0.0370401 0 - 0.0803724 2024-01-11T10:27:30.181 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 605 16 374 358 0.0369789 0 - 0.0803724 2024-01-11T10:27:31.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 606 16 374 358 0.0369179 0 - 0.0803724 2024-01-11T10:27:32.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 607 16 374 358 0.036857 0 - 0.0803724 2024-01-11T10:27:33.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 608 16 374 358 0.0367964 0 - 0.0803724 2024-01-11T10:27:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:27:33.808 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1433, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 780, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 749, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2401, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-01-11T10:27:33.808 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1301, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1433, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 780, in force_cancel_recovery self.force_recovery() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 749, in force_recovery j = self.ceph_manager.get_pgids_to_force(backfill) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2401, in get_pgids_to_force j = self.get_pg_stats(); File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-01-11T10:27:34.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 609 16 374 358 0.036736 0 - 0.0803724 2024-01-11T10:27:35.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 610 16 374 358 0.0366758 0 - 0.0803724 2024-01-11T10:27:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:27:35.282 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:27:36.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 611 16 374 358 0.0366158 0 - 0.0803724 2024-01-11T10:27:37.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 612 16 374 358 0.0365559 0 - 0.0803724 2024-01-11T10:27:37.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:27:37.402 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:27:37.423+0000 7fe9bf51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:27:38.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 613 16 374 358 0.0364963 0 - 0.0803724 2024-01-11T10:27:39.182 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 614 16 374 358 0.0364369 0 - 0.0803724 2024-01-11T10:27:40.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 615 16 374 358 0.0363776 0 - 0.0803724 2024-01-11T10:27:41.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 616 16 374 358 0.0363185 0 - 0.0803724 2024-01-11T10:27:42.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 617 16 374 358 0.0362597 0 - 0.0803724 2024-01-11T10:27:43.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 618 16 374 358 0.036201 0 - 0.0803724 2024-01-11T10:27:44.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 619 16 374 358 0.0361425 0 - 0.0803724 2024-01-11T10:27:45.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:27:45.206125+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:27:45.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:27:45.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 620 16 374 358 0.0360842 0 - 0.0803724 2024-01-11T10:27:46.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 621 16 374 358 0.0360261 0 - 0.0803724 2024-01-11T10:27:47.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 622 16 374 358 0.0359682 0 - 0.0803724 2024-01-11T10:27:48.183 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 623 16 374 358 0.0359105 0 - 0.0803724 2024-01-11T10:27:49.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 624 16 374 358 0.0358529 0 - 0.0803724 2024-01-11T10:27:50.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 625 16 374 358 0.0357956 0 - 0.0803724 2024-01-11T10:27:51.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 626 16 374 358 0.0357384 0 - 0.0803724 2024-01-11T10:27:52.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 627 16 374 358 0.0356814 0 - 0.0803724 2024-01-11T10:27:53.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 628 16 374 358 0.0356246 0 - 0.0803724 2024-01-11T10:27:54.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 629 16 374 358 0.0355679 0 - 0.0803724 2024-01-11T10:27:55.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 630 16 374 358 0.0355115 0 - 0.0803724 2024-01-11T10:27:56.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 631 16 374 358 0.0354552 0 - 0.0803724 2024-01-11T10:27:57.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 632 16 374 358 0.0353991 0 - 0.0803724 2024-01-11T10:27:58.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 633 16 374 358 0.0353432 0 - 0.0803724 2024-01-11T10:27:59.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 634 16 374 358 0.0352874 0 - 0.0803724 2024-01-11T10:28:00.184 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 635 16 374 358 0.0352319 0 - 0.0803724 2024-01-11T10:28:01.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 636 16 374 358 0.0351765 0 - 0.0803724 2024-01-11T10:28:02.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 637 16 374 358 0.0351213 0 - 0.0803724 2024-01-11T10:28:03.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 638 16 374 358 0.0350662 0 - 0.0803724 2024-01-11T10:28:04.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 639 16 374 358 0.0350113 0 - 0.0803724 2024-01-11T10:28:05.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:28:05.208257+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:28:05.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:28:05.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 640 16 374 358 0.0349566 0 - 0.0803724 2024-01-11T10:28:06.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 641 16 374 358 0.0349021 0 - 0.0803724 2024-01-11T10:28:07.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 642 16 374 358 0.0348477 0 - 0.0803724 2024-01-11T10:28:08.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 643 16 374 358 0.0347935 0 - 0.0803724 2024-01-11T10:28:09.185 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 644 16 374 358 0.0347395 0 - 0.0803724 2024-01-11T10:28:10.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 645 16 374 358 0.0346856 0 - 0.0803724 2024-01-11T10:28:11.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 646 16 374 358 0.034632 0 - 0.0803724 2024-01-11T10:28:12.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 647 16 374 358 0.0345784 0 - 0.0803724 2024-01-11T10:28:13.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 648 16 374 358 0.0345251 0 - 0.0803724 2024-01-11T10:28:14.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 649 16 374 358 0.0344719 0 - 0.0803724 2024-01-11T10:28:15.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 650 16 374 358 0.0344188 0 - 0.0803724 2024-01-11T10:28:16.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 651 16 374 358 0.034366 0 - 0.0803724 2024-01-11T10:28:17.186 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 652 16 374 358 0.0343133 0 - 0.0803724 2024-01-11T10:28:18.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 653 16 374 358 0.0342607 0 - 0.0803724 2024-01-11T10:28:19.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 654 16 374 358 0.0342083 0 - 0.0803724 2024-01-11T10:28:20.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 655 16 374 358 0.0341561 0 - 0.0803724 2024-01-11T10:28:21.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 656 16 374 358 0.034104 0 - 0.0803724 2024-01-11T10:28:22.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 657 16 374 358 0.0340521 0 - 0.0803724 2024-01-11T10:28:23.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 658 16 374 358 0.0340004 0 - 0.0803724 2024-01-11T10:28:24.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 659 16 374 358 0.0339488 0 - 0.0803724 2024-01-11T10:28:25.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:28:25.210496+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:28:25.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:28:25.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 660 16 374 358 0.0338973 0 - 0.0803724 2024-01-11T10:28:26.187 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 661 16 374 358 0.0338461 0 - 0.0803724 2024-01-11T10:28:27.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 662 16 374 358 0.0337949 0 - 0.0803724 2024-01-11T10:28:28.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 663 16 374 358 0.033744 0 - 0.0803724 2024-01-11T10:28:29.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 664 16 374 358 0.0336931 0 - 0.0803724 2024-01-11T10:28:30.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 665 16 374 358 0.0336425 0 - 0.0803724 2024-01-11T10:28:31.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 666 16 374 358 0.033592 0 - 0.0803724 2024-01-11T10:28:32.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 667 16 374 358 0.0335416 0 - 0.0803724 2024-01-11T10:28:33.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 668 16 374 358 0.0334914 0 - 0.0803724 2024-01-11T10:28:34.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 669 16 374 358 0.0334413 0 - 0.0803724 2024-01-11T10:28:35.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 670 16 374 358 0.0333914 0 - 0.0803724 2024-01-11T10:28:36.188 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 671 16 374 358 0.0333417 0 - 0.0803724 2024-01-11T10:28:37.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 672 16 374 358 0.033292 0 - 0.0803724 2024-01-11T10:28:38.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 673 16 374 358 0.0332426 0 - 0.0803724 2024-01-11T10:28:39.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 674 16 374 358 0.0331933 0 - 0.0803724 2024-01-11T10:28:40.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 675 16 374 358 0.0331441 0 - 0.0803724 2024-01-11T10:28:41.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 676 16 374 358 0.0330951 0 - 0.0803724 2024-01-11T10:28:42.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 677 16 374 358 0.0330462 0 - 0.0803724 2024-01-11T10:28:43.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 678 16 374 358 0.0329974 0 - 0.0803724 2024-01-11T10:28:44.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 679 16 374 358 0.0329488 0 - 0.0803724 2024-01-11T10:28:45.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:28:45.212694+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:28:45.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:28:45.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 680 16 374 358 0.0329004 0 - 0.0803724 2024-01-11T10:28:46.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 681 16 374 358 0.0328521 0 - 0.0803724 2024-01-11T10:28:47.189 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 682 16 374 358 0.0328039 0 - 0.0803724 2024-01-11T10:28:48.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 683 16 374 358 0.0327559 0 - 0.0803724 2024-01-11T10:28:49.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 684 16 374 358 0.032708 0 - 0.0803724 2024-01-11T10:28:50.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 685 16 374 358 0.0326602 0 - 0.0803724 2024-01-11T10:28:51.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 686 16 374 358 0.0326126 0 - 0.0803724 2024-01-11T10:28:52.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 687 16 374 358 0.0325652 0 - 0.0803724 2024-01-11T10:28:53.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 688 16 374 358 0.0325178 0 - 0.0803724 2024-01-11T10:28:54.190 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 689 16 374 358 0.0324706 0 - 0.0803724 2024-01-11T10:28:55.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 690 16 374 358 0.0324236 0 - 0.0803724 2024-01-11T10:28:56.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 691 16 374 358 0.0323766 0 - 0.0803724 2024-01-11T10:28:57.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 692 16 374 358 0.0323299 0 - 0.0803724 2024-01-11T10:28:58.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 693 16 374 358 0.0322832 0 - 0.0803724 2024-01-11T10:28:59.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 694 16 374 358 0.0322367 0 - 0.0803724 2024-01-11T10:29:00.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 695 16 374 358 0.0321903 0 - 0.0803724 2024-01-11T10:29:01.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 696 16 374 358 0.0321441 0 - 0.0803724 2024-01-11T10:29:02.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 697 16 374 358 0.0320979 0 - 0.0803724 2024-01-11T10:29:03.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 698 16 374 358 0.032052 0 - 0.0803724 2024-01-11T10:29:04.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 699 16 374 358 0.0320061 0 - 0.0803724 2024-01-11T10:29:05.191 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:29:05.214783+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:29:05.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:29:05.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 700 16 374 358 0.0319604 0 - 0.0803724 2024-01-11T10:29:06.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 701 16 374 358 0.0319148 0 - 0.0803724 2024-01-11T10:29:07.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 702 16 374 358 0.0318693 0 - 0.0803724 2024-01-11T10:29:08.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 703 16 374 358 0.031824 0 - 0.0803724 2024-01-11T10:29:09.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 704 16 374 358 0.0317788 0 - 0.0803724 2024-01-11T10:29:10.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 705 16 374 358 0.0317337 0 - 0.0803724 2024-01-11T10:29:11.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 706 16 374 358 0.0316888 0 - 0.0803724 2024-01-11T10:29:12.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 707 16 374 358 0.0316439 0 - 0.0803724 2024-01-11T10:29:13.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 708 16 374 358 0.0315993 0 - 0.0803724 2024-01-11T10:29:14.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 709 16 374 358 0.0315547 0 - 0.0803724 2024-01-11T10:29:15.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 710 16 374 358 0.0315102 0 - 0.0803724 2024-01-11T10:29:16.192 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 711 16 374 358 0.0314659 0 - 0.0803724 2024-01-11T10:29:17.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 712 16 374 358 0.0314217 0 - 0.0803724 2024-01-11T10:29:18.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 713 16 374 358 0.0313777 0 - 0.0803724 2024-01-11T10:29:19.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 714 16 374 358 0.0313337 0 - 0.0803724 2024-01-11T10:29:20.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 715 16 374 358 0.0312899 0 - 0.0803724 2024-01-11T10:29:21.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 716 16 374 358 0.0312462 0 - 0.0803724 2024-01-11T10:29:22.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 717 16 374 358 0.0312026 0 - 0.0803724 2024-01-11T10:29:23.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 718 16 374 358 0.0311592 0 - 0.0803724 2024-01-11T10:29:24.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 719 16 374 358 0.0311158 0 - 0.0803724 2024-01-11T10:29:25.193 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:29:25.216875+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:29:25.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:29:25.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 720 16 374 358 0.0310726 0 - 0.0803724 2024-01-11T10:29:26.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 721 16 374 358 0.0310295 0 - 0.0803724 2024-01-11T10:29:27.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 722 16 374 358 0.0309865 0 - 0.0803724 2024-01-11T10:29:28.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 723 16 374 358 0.0309437 0 - 0.0803724 2024-01-11T10:29:29.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 724 16 374 358 0.0309009 0 - 0.0803724 2024-01-11T10:29:30.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 725 16 374 358 0.0308583 0 - 0.0803724 2024-01-11T10:29:31.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 726 16 374 358 0.0308158 0 - 0.0803724 2024-01-11T10:29:32.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 727 16 374 358 0.0307734 0 - 0.0803724 2024-01-11T10:29:33.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 728 16 374 358 0.0307312 0 - 0.0803724 2024-01-11T10:29:34.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 729 16 374 358 0.030689 0 - 0.0803724 2024-01-11T10:29:35.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 730 16 374 358 0.030647 0 - 0.0803724 2024-01-11T10:29:36.194 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 731 16 374 358 0.030605 0 - 0.0803724 2024-01-11T10:29:37.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 732 16 374 358 0.0305632 0 - 0.0803724 2024-01-11T10:29:37.320 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:29:37.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:29:38.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 733 16 374 358 0.0305215 0 - 0.0803724 2024-01-11T10:29:39.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 734 16 374 358 0.03048 0 - 0.0803724 2024-01-11T10:29:39.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:29:39.442 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:29:39.464+0000 7fbea522d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:29:40.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 735 16 374 358 0.0304385 0 - 0.0803724 2024-01-11T10:29:41.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 736 16 374 358 0.0303971 0 - 0.0803724 2024-01-11T10:29:42.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 737 16 374 358 0.0303559 0 - 0.0803724 2024-01-11T10:29:43.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 738 16 374 358 0.0303147 0 - 0.0803724 2024-01-11T10:29:44.195 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 739 16 374 358 0.0302737 0 - 0.0803724 2024-01-11T10:29:45.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:29:45.219139+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:29:45.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:29:45.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 740 16 374 358 0.0302328 0 - 0.0803724 2024-01-11T10:29:46.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 741 16 374 358 0.030192 0 - 0.0803724 2024-01-11T10:29:47.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 742 16 374 358 0.0301513 0 - 0.0803724 2024-01-11T10:29:48.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 743 16 374 358 0.0301107 0 - 0.0803724 2024-01-11T10:29:49.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 744 16 374 358 0.0300703 0 - 0.0803724 2024-01-11T10:29:50.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 745 16 374 358 0.0300299 0 - 0.0803724 2024-01-11T10:29:51.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 746 16 374 358 0.0299897 0 - 0.0803724 2024-01-11T10:29:52.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 747 16 374 358 0.0299495 0 - 0.0803724 2024-01-11T10:29:53.196 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 748 16 374 358 0.0299095 0 - 0.0803724 2024-01-11T10:29:54.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 749 16 374 358 0.0298695 0 - 0.0803724 2024-01-11T10:29:55.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 750 16 374 358 0.0298297 0 - 0.0803724 2024-01-11T10:29:56.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 751 16 374 358 0.02979 0 - 0.0803724 2024-01-11T10:29:57.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 752 16 374 358 0.0297504 0 - 0.0803724 2024-01-11T10:29:58.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 753 16 374 358 0.0297109 0 - 0.0803724 2024-01-11T10:29:59.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 754 16 374 358 0.0296715 0 - 0.0803724 2024-01-11T10:30:00.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 755 16 374 358 0.0296322 0 - 0.0803724 2024-01-11T10:30:01.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 756 16 374 358 0.029593 0 - 0.0803724 2024-01-11T10:30:02.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 757 16 374 358 0.0295539 0 - 0.0803724 2024-01-11T10:30:03.197 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 758 16 374 358 0.0295149 0 - 0.0803724 2024-01-11T10:30:04.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 759 16 374 358 0.029476 0 - 0.0803724 2024-01-11T10:30:05.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:30:05.221295+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:30:05.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:30:05.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 760 16 374 358 0.0294372 0 - 0.0803724 2024-01-11T10:30:06.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 761 16 374 358 0.0293985 0 - 0.0803724 2024-01-11T10:30:07.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 762 16 374 358 0.02936 0 - 0.0803724 2024-01-11T10:30:08.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 763 16 374 358 0.0293215 0 - 0.0803724 2024-01-11T10:30:09.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 764 16 374 358 0.0292831 0 - 0.0803724 2024-01-11T10:30:10.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 765 16 374 358 0.0292448 0 - 0.0803724 2024-01-11T10:30:11.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 766 16 374 358 0.0292067 0 - 0.0803724 2024-01-11T10:30:12.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 767 16 374 358 0.0291686 0 - 0.0803724 2024-01-11T10:30:13.198 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 768 16 374 358 0.0291306 0 - 0.0803724 2024-01-11T10:30:14.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 769 16 374 358 0.0290927 0 - 0.0803724 2024-01-11T10:30:15.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 770 16 374 358 0.0290549 0 - 0.0803724 2024-01-11T10:30:16.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 771 16 374 358 0.0290172 0 - 0.0803724 2024-01-11T10:30:17.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 772 16 374 358 0.0289797 0 - 0.0803724 2024-01-11T10:30:18.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 773 16 374 358 0.0289422 0 - 0.0803724 2024-01-11T10:30:19.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 774 16 374 358 0.0289048 0 - 0.0803724 2024-01-11T10:30:20.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 775 16 374 358 0.0288675 0 - 0.0803724 2024-01-11T10:30:21.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 776 16 374 358 0.0288303 0 - 0.0803724 2024-01-11T10:30:22.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 777 16 374 358 0.0287932 0 - 0.0803724 2024-01-11T10:30:23.199 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 778 16 374 358 0.0287562 0 - 0.0803724 2024-01-11T10:30:24.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 779 16 374 358 0.0287193 0 - 0.0803724 2024-01-11T10:30:25.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:30:25.223468+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:30:25.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:30:25.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 780 16 374 358 0.0286824 0 - 0.0803724 2024-01-11T10:30:26.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 781 16 374 358 0.0286457 0 - 0.0803724 2024-01-11T10:30:27.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 782 16 374 358 0.0286091 0 - 0.0803724 2024-01-11T10:30:28.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 783 16 374 358 0.0285725 0 - 0.0803724 2024-01-11T10:30:29.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 784 16 374 358 0.0285361 0 - 0.0803724 2024-01-11T10:30:30.200 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 785 16 374 358 0.0284997 0 - 0.0803724 2024-01-11T10:30:31.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 786 16 374 358 0.0284635 0 - 0.0803724 2024-01-11T10:30:32.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 787 16 374 358 0.0284273 0 - 0.0803724 2024-01-11T10:30:33.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 788 16 374 358 0.0283912 0 - 0.0803724 2024-01-11T10:30:34.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 789 16 374 358 0.0283553 0 - 0.0803724 2024-01-11T10:30:35.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 790 16 374 358 0.0283194 0 - 0.0803724 2024-01-11T10:30:36.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 791 16 374 358 0.0282836 0 - 0.0803724 2024-01-11T10:30:37.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 792 16 374 358 0.0282479 0 - 0.0803724 2024-01-11T10:30:38.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 793 16 374 358 0.0282122 0 - 0.0803724 2024-01-11T10:30:39.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 794 16 374 358 0.0281767 0 - 0.0803724 2024-01-11T10:30:40.201 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 795 16 374 358 0.0281413 0 - 0.0803724 2024-01-11T10:30:41.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 796 16 374 358 0.0281059 0 - 0.0803724 2024-01-11T10:30:42.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 797 16 374 358 0.0280706 0 - 0.0803724 2024-01-11T10:30:43.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 798 16 374 358 0.0280355 0 - 0.0803724 2024-01-11T10:30:44.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 799 16 374 358 0.0280004 0 - 0.0803724 2024-01-11T10:30:45.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:30:45.225610+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:30:45.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:30:45.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 800 16 374 358 0.0279654 0 - 0.0803724 2024-01-11T10:30:46.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 801 16 374 358 0.0279305 0 - 0.0803724 2024-01-11T10:30:47.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 802 16 374 358 0.0278956 0 - 0.0803724 2024-01-11T10:30:48.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 803 16 374 358 0.0278609 0 - 0.0803724 2024-01-11T10:30:49.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 804 16 374 358 0.0278263 0 - 0.0803724 2024-01-11T10:30:50.202 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 805 16 374 358 0.0277917 0 - 0.0803724 2024-01-11T10:30:51.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 806 16 374 358 0.0277572 0 - 0.0803724 2024-01-11T10:30:52.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 807 16 374 358 0.0277228 0 - 0.0803724 2024-01-11T10:30:53.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 808 16 374 358 0.0276885 0 - 0.0803724 2024-01-11T10:30:54.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 809 16 374 358 0.0276543 0 - 0.0803724 2024-01-11T10:30:55.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 810 16 374 358 0.0276201 0 - 0.0803724 2024-01-11T10:30:56.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 811 16 374 358 0.0275861 0 - 0.0803724 2024-01-11T10:30:57.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 812 16 374 358 0.0275521 0 - 0.0803724 2024-01-11T10:30:58.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 813 16 374 358 0.0275182 0 - 0.0803724 2024-01-11T10:30:59.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 814 16 374 358 0.0274844 0 - 0.0803724 2024-01-11T10:31:00.203 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 815 16 374 358 0.0274507 0 - 0.0803724 2024-01-11T10:31:01.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 816 16 374 358 0.027417 0 - 0.0803724 2024-01-11T10:31:02.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 817 16 374 358 0.0273835 0 - 0.0803724 2024-01-11T10:31:03.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 818 16 374 358 0.02735 0 - 0.0803724 2024-01-11T10:31:04.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 819 16 374 358 0.0273166 0 - 0.0803724 2024-01-11T10:31:05.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:31:05.227837+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:31:05.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:31:05.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 820 16 374 358 0.0272833 0 - 0.0803724 2024-01-11T10:31:06.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 821 16 374 358 0.0272501 0 - 0.0803724 2024-01-11T10:31:07.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 822 16 374 358 0.0272169 0 - 0.0803724 2024-01-11T10:31:08.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 823 16 374 358 0.0271839 0 - 0.0803724 2024-01-11T10:31:09.204 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 824 16 374 358 0.0271509 0 - 0.0803724 2024-01-11T10:31:10.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 825 16 374 358 0.027118 0 - 0.0803724 2024-01-11T10:31:11.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 826 16 374 358 0.0270851 0 - 0.0803724 2024-01-11T10:31:12.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 827 16 374 358 0.0270524 0 - 0.0803724 2024-01-11T10:31:13.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 828 16 374 358 0.0270197 0 - 0.0803724 2024-01-11T10:31:14.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 829 16 374 358 0.0269871 0 - 0.0803724 2024-01-11T10:31:15.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 830 16 374 358 0.0269546 0 - 0.0803724 2024-01-11T10:31:16.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 831 16 374 358 0.0269222 0 - 0.0803724 2024-01-11T10:31:17.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 832 16 374 358 0.0268898 0 - 0.0803724 2024-01-11T10:31:18.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 833 16 374 358 0.0268575 0 - 0.0803724 2024-01-11T10:31:19.205 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 834 16 374 358 0.0268253 0 - 0.0803724 2024-01-11T10:31:20.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 835 16 374 358 0.0267932 0 - 0.0803724 2024-01-11T10:31:21.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 836 16 374 358 0.0267611 0 - 0.0803724 2024-01-11T10:31:22.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 837 16 374 358 0.0267292 0 - 0.0803724 2024-01-11T10:31:23.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 838 16 374 358 0.0266973 0 - 0.0803724 2024-01-11T10:31:24.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 839 16 374 358 0.0266655 0 - 0.0803724 2024-01-11T10:31:25.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:31:25.230051+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:31:25.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:31:25.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 840 16 374 358 0.0266337 0 - 0.0803724 2024-01-11T10:31:26.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 841 16 374 358 0.026602 0 - 0.0803724 2024-01-11T10:31:27.206 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 842 16 374 358 0.0265704 0 - 0.0803724 2024-01-11T10:31:28.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 843 16 374 358 0.0265389 0 - 0.0803724 2024-01-11T10:31:29.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 844 16 374 358 0.0265075 0 - 0.0803724 2024-01-11T10:31:30.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 845 16 374 358 0.0264761 0 - 0.0803724 2024-01-11T10:31:31.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 846 16 374 358 0.0264448 0 - 0.0803724 2024-01-11T10:31:32.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 847 16 374 358 0.0264136 0 - 0.0803724 2024-01-11T10:31:33.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 848 16 374 358 0.0263824 0 - 0.0803724 2024-01-11T10:31:34.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 849 16 374 358 0.0263514 0 - 0.0803724 2024-01-11T10:31:35.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 850 16 374 358 0.0263204 0 - 0.0803724 2024-01-11T10:31:36.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 851 16 374 358 0.0262894 0 - 0.0803724 2024-01-11T10:31:37.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 852 16 374 358 0.0262586 0 - 0.0803724 2024-01-11T10:31:38.207 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 853 16 374 358 0.0262278 0 - 0.0803724 2024-01-11T10:31:39.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 854 16 374 358 0.0261971 0 - 0.0803724 2024-01-11T10:31:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:31:39.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:31:40.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 855 16 374 358 0.0261665 0 - 0.0803724 2024-01-11T10:31:41.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 856 16 374 358 0.0261359 0 - 0.0803724 2024-01-11T10:31:41.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:31:41.478 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:31:41.499+0000 7febdfc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:31:42.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 857 16 374 358 0.0261054 0 - 0.0803724 2024-01-11T10:31:43.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 858 16 374 358 0.026075 0 - 0.0803724 2024-01-11T10:31:44.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 859 16 374 358 0.0260446 0 - 0.0803724 2024-01-11T10:31:45.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:31:45.232163+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:31:45.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:31:45.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 860 16 374 358 0.0260143 0 - 0.0803724 2024-01-11T10:31:46.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 861 16 374 358 0.0259841 0 - 0.0803724 2024-01-11T10:31:47.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 862 16 374 358 0.025954 0 - 0.0803724 2024-01-11T10:31:48.208 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 863 16 374 358 0.0259239 0 - 0.0803724 2024-01-11T10:31:49.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 864 16 374 358 0.0258939 0 - 0.0803724 2024-01-11T10:31:50.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 865 16 374 358 0.025864 0 - 0.0803724 2024-01-11T10:31:51.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 866 16 374 358 0.0258341 0 - 0.0803724 2024-01-11T10:31:52.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 867 16 374 358 0.0258043 0 - 0.0803724 2024-01-11T10:31:53.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 868 16 374 358 0.0257746 0 - 0.0803724 2024-01-11T10:31:54.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 869 16 374 358 0.0257449 0 - 0.0803724 2024-01-11T10:31:55.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 870 16 374 358 0.0257153 0 - 0.0803724 2024-01-11T10:31:56.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 871 16 374 358 0.0256858 0 - 0.0803724 2024-01-11T10:31:57.209 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 872 16 374 358 0.0256563 0 - 0.0803724 2024-01-11T10:31:58.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 873 16 374 358 0.0256269 0 - 0.0803724 2024-01-11T10:31:59.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 874 16 374 358 0.0255976 0 - 0.0803724 2024-01-11T10:32:00.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 875 16 374 358 0.0255684 0 - 0.0803724 2024-01-11T10:32:01.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 876 16 374 358 0.0255392 0 - 0.0803724 2024-01-11T10:32:02.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 877 16 374 358 0.0255101 0 - 0.0803724 2024-01-11T10:32:03.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 878 16 374 358 0.025481 0 - 0.0803724 2024-01-11T10:32:04.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 879 16 374 358 0.025452 0 - 0.0803724 2024-01-11T10:32:05.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:32:05.234362+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:32:05.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:32:05.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 880 16 374 358 0.0254231 0 - 0.0803724 2024-01-11T10:32:06.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 881 16 374 358 0.0253942 0 - 0.0803724 2024-01-11T10:32:07.210 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 882 16 374 358 0.0253655 0 - 0.0803724 2024-01-11T10:32:08.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 883 16 374 358 0.0253367 0 - 0.0803724 2024-01-11T10:32:09.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 884 16 374 358 0.0253081 0 - 0.0803724 2024-01-11T10:32:10.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 885 16 374 358 0.0252795 0 - 0.0803724 2024-01-11T10:32:11.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 886 16 374 358 0.0252509 0 - 0.0803724 2024-01-11T10:32:12.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 887 16 374 358 0.0252225 0 - 0.0803724 2024-01-11T10:32:13.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 888 16 374 358 0.0251941 0 - 0.0803724 2024-01-11T10:32:14.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 889 16 374 358 0.0251657 0 - 0.0803724 2024-01-11T10:32:15.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 890 16 374 358 0.0251375 0 - 0.0803724 2024-01-11T10:32:16.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 891 16 374 358 0.0251092 0 - 0.0803724 2024-01-11T10:32:17.211 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 892 16 374 358 0.0250811 0 - 0.0803724 2024-01-11T10:32:18.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 893 16 374 358 0.025053 0 - 0.0803724 2024-01-11T10:32:19.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 894 16 374 358 0.025025 0 - 0.0803724 2024-01-11T10:32:20.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 895 16 374 358 0.024997 0 - 0.0803724 2024-01-11T10:32:21.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 896 16 374 358 0.0249691 0 - 0.0803724 2024-01-11T10:32:22.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 897 16 374 358 0.0249413 0 - 0.0803724 2024-01-11T10:32:23.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 898 16 374 358 0.0249135 0 - 0.0803724 2024-01-11T10:32:24.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 899 16 374 358 0.0248858 0 - 0.0803724 2024-01-11T10:32:25.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:32:25.236328+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:32:25.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:32:25.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 900 16 374 358 0.0248582 0 - 0.0803724 2024-01-11T10:32:26.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 901 16 374 358 0.0248306 0 - 0.0803724 2024-01-11T10:32:27.212 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 902 16 374 358 0.024803 0 - 0.0803724 2024-01-11T10:32:28.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 903 16 374 358 0.0247756 0 - 0.0803724 2024-01-11T10:32:29.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 904 16 374 358 0.0247482 0 - 0.0803724 2024-01-11T10:32:30.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 905 16 374 358 0.0247208 0 - 0.0803724 2024-01-11T10:32:31.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 906 16 374 358 0.0246935 0 - 0.0803724 2024-01-11T10:32:32.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 907 16 374 358 0.0246663 0 - 0.0803724 2024-01-11T10:32:33.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 908 16 374 358 0.0246391 0 - 0.0803724 2024-01-11T10:32:34.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 909 16 374 358 0.024612 0 - 0.0803724 2024-01-11T10:32:35.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 910 16 374 358 0.024585 0 - 0.0803724 2024-01-11T10:32:36.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 911 16 374 358 0.024558 0 - 0.0803724 2024-01-11T10:32:37.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 912 16 374 358 0.0245311 0 - 0.0803724 2024-01-11T10:32:38.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 913 16 374 358 0.0245042 0 - 0.0803724 2024-01-11T10:32:39.213 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 914 16 374 358 0.0244774 0 - 0.0803724 2024-01-11T10:32:40.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 915 16 374 358 0.0244507 0 - 0.0803724 2024-01-11T10:32:41.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 916 16 374 358 0.024424 0 - 0.0803724 2024-01-11T10:32:42.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 917 16 374 358 0.0243973 0 - 0.0803724 2024-01-11T10:32:43.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 918 16 374 358 0.0243708 0 - 0.0803724 2024-01-11T10:32:44.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 919 16 374 358 0.0243442 0 - 0.0803724 2024-01-11T10:32:45.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:32:45.238247+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:32:45.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:32:45.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 920 16 374 358 0.0243178 0 - 0.0803724 2024-01-11T10:32:46.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 921 16 374 358 0.0242914 0 - 0.0803724 2024-01-11T10:32:47.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 922 16 374 358 0.024265 0 - 0.0803724 2024-01-11T10:32:48.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 923 16 374 358 0.0242387 0 - 0.0803724 2024-01-11T10:32:49.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 924 16 374 358 0.0242125 0 - 0.0803724 2024-01-11T10:32:50.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 925 16 374 358 0.0241863 0 - 0.0803724 2024-01-11T10:32:51.214 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 926 16 374 358 0.0241602 0 - 0.0803724 2024-01-11T10:32:52.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 927 16 374 358 0.0241341 0 - 0.0803724 2024-01-11T10:32:53.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 928 16 374 358 0.0241081 0 - 0.0803724 2024-01-11T10:32:54.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 929 16 374 358 0.0240822 0 - 0.0803724 2024-01-11T10:32:55.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 930 16 374 358 0.0240563 0 - 0.0803724 2024-01-11T10:32:56.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 931 16 374 358 0.0240305 0 - 0.0803724 2024-01-11T10:32:57.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 932 16 374 358 0.0240047 0 - 0.0803724 2024-01-11T10:32:58.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 933 16 374 358 0.0239789 0 - 0.0803724 2024-01-11T10:32:59.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 934 16 374 358 0.0239533 0 - 0.0803724 2024-01-11T10:33:00.215 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 935 16 374 358 0.0239277 0 - 0.0803724 2024-01-11T10:33:01.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 936 16 374 358 0.0239021 0 - 0.0803724 2024-01-11T10:33:02.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 937 16 374 358 0.0238766 0 - 0.0803724 2024-01-11T10:33:03.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 938 16 374 358 0.0238511 0 - 0.0803724 2024-01-11T10:33:04.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 939 16 374 358 0.0238257 0 - 0.0803724 2024-01-11T10:33:05.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:33:05.240259+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:33:05.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:33:05.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 940 16 374 358 0.0238004 0 - 0.0803724 2024-01-11T10:33:06.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 941 16 374 358 0.0237751 0 - 0.0803724 2024-01-11T10:33:07.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 942 16 374 358 0.0237499 0 - 0.0803724 2024-01-11T10:33:08.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 943 16 374 358 0.0237247 0 - 0.0803724 2024-01-11T10:33:09.216 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 944 16 374 358 0.0236995 0 - 0.0803724 2024-01-11T10:33:10.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 945 16 374 358 0.0236745 0 - 0.0803724 2024-01-11T10:33:11.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 946 16 374 358 0.0236494 0 - 0.0803724 2024-01-11T10:33:12.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 947 16 374 358 0.0236245 0 - 0.0803724 2024-01-11T10:33:13.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 948 16 374 358 0.0235995 0 - 0.0803724 2024-01-11T10:33:14.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 949 16 374 358 0.0235747 0 - 0.0803724 2024-01-11T10:33:15.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 950 16 374 358 0.0235499 0 - 0.0803724 2024-01-11T10:33:16.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 951 16 374 358 0.0235251 0 - 0.0803724 2024-01-11T10:33:17.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 952 16 374 358 0.0235004 0 - 0.0803724 2024-01-11T10:33:18.217 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 953 16 374 358 0.0234757 0 - 0.0803724 2024-01-11T10:33:19.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 954 16 374 358 0.0234511 0 - 0.0803724 2024-01-11T10:33:20.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 955 16 374 358 0.0234266 0 - 0.0803724 2024-01-11T10:33:21.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 956 16 374 358 0.0234021 0 - 0.0803724 2024-01-11T10:33:22.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 957 16 374 358 0.0233776 0 - 0.0803724 2024-01-11T10:33:23.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 958 16 374 358 0.0233532 0 - 0.0803724 2024-01-11T10:33:24.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 959 16 374 358 0.0233288 0 - 0.0803724 2024-01-11T10:33:25.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:33:25.242422+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:33:25.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:33:25.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 960 16 374 358 0.0233045 0 - 0.0803724 2024-01-11T10:33:26.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 961 16 374 358 0.0232803 0 - 0.0803724 2024-01-11T10:33:27.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 962 16 374 358 0.0232561 0 - 0.0803724 2024-01-11T10:33:28.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 963 16 374 358 0.0232319 0 - 0.0803724 2024-01-11T10:33:29.218 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 964 16 374 358 0.0232078 0 - 0.0803724 2024-01-11T10:33:30.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 965 16 374 358 0.0231838 0 - 0.0803724 2024-01-11T10:33:31.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 966 16 374 358 0.0231598 0 - 0.0803724 2024-01-11T10:33:32.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 967 16 374 358 0.0231358 0 - 0.0803724 2024-01-11T10:33:33.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 968 16 374 358 0.0231119 0 - 0.0803724 2024-01-11T10:33:34.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 969 16 374 358 0.0230881 0 - 0.0803724 2024-01-11T10:33:35.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 970 16 374 358 0.0230643 0 - 0.0803724 2024-01-11T10:33:36.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 971 16 374 358 0.0230405 0 - 0.0803724 2024-01-11T10:33:37.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 972 16 374 358 0.0230168 0 - 0.0803724 2024-01-11T10:33:38.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 973 16 374 358 0.0229932 0 - 0.0803724 2024-01-11T10:33:39.219 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 974 16 374 358 0.0229696 0 - 0.0803724 2024-01-11T10:33:40.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 975 16 374 358 0.022946 0 - 0.0803724 2024-01-11T10:33:41.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 976 16 374 358 0.0229225 0 - 0.0803724 2024-01-11T10:33:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:33:41.398 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:33:42.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 977 16 374 358 0.022899 0 - 0.0803724 2024-01-11T10:33:43.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 978 16 374 358 0.0228756 0 - 0.0803724 2024-01-11T10:33:43.399 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:33:43.518 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:33:43.540+0000 7f0d0c11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:33:44.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 979 16 374 358 0.0228523 0 - 0.0803724 2024-01-11T10:33:45.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:33:45.244477+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:33:45.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:33:45.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 980 16 374 358 0.0228289 0 - 0.0803724 2024-01-11T10:33:46.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 981 16 374 358 0.0228057 0 - 0.0803724 2024-01-11T10:33:47.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 982 16 374 358 0.0227825 0 - 0.0803724 2024-01-11T10:33:48.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 983 16 374 358 0.0227593 0 - 0.0803724 2024-01-11T10:33:49.220 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 984 16 374 358 0.0227361 0 - 0.0803724 2024-01-11T10:33:50.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 985 16 374 358 0.0227131 0 - 0.0803724 2024-01-11T10:33:51.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 986 16 374 358 0.02269 0 - 0.0803724 2024-01-11T10:33:52.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 987 16 374 358 0.022667 0 - 0.0803724 2024-01-11T10:33:53.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 988 16 374 358 0.0226441 0 - 0.0803724 2024-01-11T10:33:54.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 989 16 374 358 0.0226212 0 - 0.0803724 2024-01-11T10:33:55.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 990 16 374 358 0.0225984 0 - 0.0803724 2024-01-11T10:33:56.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 991 16 374 358 0.0225756 0 - 0.0803724 2024-01-11T10:33:57.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 992 16 374 358 0.0225528 0 - 0.0803724 2024-01-11T10:33:58.221 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 993 16 374 358 0.0225301 0 - 0.0803724 2024-01-11T10:33:59.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 994 16 374 358 0.0225074 0 - 0.0803724 2024-01-11T10:34:00.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 995 16 374 358 0.0224848 0 - 0.0803724 2024-01-11T10:34:01.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 996 16 374 358 0.0224622 0 - 0.0803724 2024-01-11T10:34:02.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 997 16 374 358 0.0224397 0 - 0.0803724 2024-01-11T10:34:03.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 998 16 374 358 0.0224172 0 - 0.0803724 2024-01-11T10:34:04.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 999 16 374 358 0.0223948 0 - 0.0803724 2024-01-11T10:34:05.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:34:05.246671+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:34:05.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:34:05.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1000 16 374 358 0.0223724 0 - 0.0803724 2024-01-11T10:34:06.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1001 16 374 358 0.02235 0 - 0.0803724 2024-01-11T10:34:07.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1002 16 374 358 0.0223277 0 - 0.0803724 2024-01-11T10:34:08.222 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1003 16 374 358 0.0223055 0 - 0.0803724 2024-01-11T10:34:09.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1004 16 374 358 0.0222832 0 - 0.0803724 2024-01-11T10:34:10.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1005 16 374 358 0.0222611 0 - 0.0803724 2024-01-11T10:34:11.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1006 16 374 358 0.0222389 0 - 0.0803724 2024-01-11T10:34:12.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1007 16 374 358 0.0222169 0 - 0.0803724 2024-01-11T10:34:13.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1008 16 374 358 0.0221948 0 - 0.0803724 2024-01-11T10:34:14.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1009 16 374 358 0.0221728 0 - 0.0803724 2024-01-11T10:34:15.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1010 16 374 358 0.0221509 0 - 0.0803724 2024-01-11T10:34:16.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1011 16 374 358 0.022129 0 - 0.0803724 2024-01-11T10:34:17.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1012 16 374 358 0.0221071 0 - 0.0803724 2024-01-11T10:34:18.223 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1013 16 374 358 0.0220853 0 - 0.0803724 2024-01-11T10:34:19.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1014 16 374 358 0.0220635 0 - 0.0803724 2024-01-11T10:34:20.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1015 16 374 358 0.0220418 0 - 0.0803724 2024-01-11T10:34:21.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1016 16 374 358 0.0220201 0 - 0.0803724 2024-01-11T10:34:22.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1017 16 374 358 0.0219984 0 - 0.0803724 2024-01-11T10:34:23.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1018 16 374 358 0.0219768 0 - 0.0803724 2024-01-11T10:34:24.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1019 16 374 358 0.0219552 0 - 0.0803724 2024-01-11T10:34:25.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:34:25.248789+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:34:25.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:34:25.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1020 16 374 358 0.0219337 0 - 0.0803724 2024-01-11T10:34:26.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1021 16 374 358 0.0219122 0 - 0.0803724 2024-01-11T10:34:27.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1022 16 374 358 0.0218908 0 - 0.0803724 2024-01-11T10:34:28.224 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1023 16 374 358 0.0218694 0 - 0.0803724 2024-01-11T10:34:29.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1024 16 374 358 0.021848 0 - 0.0803724 2024-01-11T10:34:30.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1025 16 374 358 0.0218267 0 - 0.0803724 2024-01-11T10:34:31.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1026 16 374 358 0.0218054 0 - 0.0803724 2024-01-11T10:34:32.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1027 16 374 358 0.0217842 0 - 0.0803724 2024-01-11T10:34:33.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1028 16 374 358 0.021763 0 - 0.0803724 2024-01-11T10:34:34.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1029 16 374 358 0.0217419 0 - 0.0803724 2024-01-11T10:34:35.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1030 16 374 358 0.0217208 0 - 0.0803724 2024-01-11T10:34:36.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1031 16 374 358 0.0216997 0 - 0.0803724 2024-01-11T10:34:37.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1032 16 374 358 0.0216787 0 - 0.0803724 2024-01-11T10:34:38.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1033 16 374 358 0.0216577 0 - 0.0803724 2024-01-11T10:34:39.225 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1034 16 374 358 0.0216367 0 - 0.0803724 2024-01-11T10:34:40.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1035 16 374 358 0.0216158 0 - 0.0803724 2024-01-11T10:34:41.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1036 16 374 358 0.021595 0 - 0.0803724 2024-01-11T10:34:42.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1037 16 374 358 0.0215741 0 - 0.0803724 2024-01-11T10:34:43.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1038 16 374 358 0.0215534 0 - 0.0803724 2024-01-11T10:34:44.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1039 16 374 358 0.0215326 0 - 0.0803724 2024-01-11T10:34:45.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:34:45.250896+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:34:45.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:34:45.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1040 16 374 358 0.0215119 0 - 0.0803724 2024-01-11T10:34:46.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1041 16 374 358 0.0214912 0 - 0.0803724 2024-01-11T10:34:47.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1042 16 374 358 0.0214706 0 - 0.0803724 2024-01-11T10:34:48.226 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1043 16 374 358 0.02145 0 - 0.0803724 2024-01-11T10:34:49.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1044 16 374 358 0.0214295 0 - 0.0803724 2024-01-11T10:34:50.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1045 16 374 358 0.021409 0 - 0.0803724 2024-01-11T10:34:51.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1046 16 374 358 0.0213885 0 - 0.0803724 2024-01-11T10:34:52.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1047 16 374 358 0.0213681 0 - 0.0803724 2024-01-11T10:34:53.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1048 16 374 358 0.0213477 0 - 0.0803724 2024-01-11T10:34:54.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1049 16 374 358 0.0213273 0 - 0.0803724 2024-01-11T10:34:55.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1050 16 374 358 0.021307 0 - 0.0803724 2024-01-11T10:34:56.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1051 16 374 358 0.0212868 0 - 0.0803724 2024-01-11T10:34:57.227 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1052 16 374 358 0.0212665 0 - 0.0803724 2024-01-11T10:34:58.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1053 16 374 358 0.0212463 0 - 0.0803724 2024-01-11T10:34:59.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1054 16 374 358 0.0212262 0 - 0.0803724 2024-01-11T10:35:00.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1055 16 374 358 0.0212061 0 - 0.0803724 2024-01-11T10:35:01.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1056 16 374 358 0.021186 0 - 0.0803724 2024-01-11T10:35:02.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1057 16 374 358 0.0211659 0 - 0.0803724 2024-01-11T10:35:03.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1058 16 374 358 0.0211459 0 - 0.0803724 2024-01-11T10:35:04.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1059 16 374 358 0.021126 0 - 0.0803724 2024-01-11T10:35:05.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:35:05.253051+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:35:05.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:35:05.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1060 16 374 358 0.021106 0 - 0.0803724 2024-01-11T10:35:06.228 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1061 16 374 358 0.0210861 0 - 0.0803724 2024-01-11T10:35:07.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1062 16 374 358 0.0210663 0 - 0.0803724 2024-01-11T10:35:08.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1063 16 374 358 0.0210465 0 - 0.0803724 2024-01-11T10:35:09.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1064 16 374 358 0.0210267 0 - 0.0803724 2024-01-11T10:35:10.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1065 16 374 358 0.0210069 0 - 0.0803724 2024-01-11T10:35:11.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1066 16 374 358 0.0209872 0 - 0.0803724 2024-01-11T10:35:12.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1067 16 374 358 0.0209676 0 - 0.0803724 2024-01-11T10:35:13.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1068 16 374 358 0.0209479 0 - 0.0803724 2024-01-11T10:35:14.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1069 16 374 358 0.0209283 0 - 0.0803724 2024-01-11T10:35:15.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1070 16 374 358 0.0209088 0 - 0.0803724 2024-01-11T10:35:16.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1071 16 374 358 0.0208893 0 - 0.0803724 2024-01-11T10:35:17.229 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1072 16 374 358 0.0208698 0 - 0.0803724 2024-01-11T10:35:18.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1073 16 374 358 0.0208503 0 - 0.0803724 2024-01-11T10:35:19.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1074 16 374 358 0.0208309 0 - 0.0803724 2024-01-11T10:35:20.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1075 16 374 358 0.0208115 0 - 0.0803724 2024-01-11T10:35:21.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1076 16 374 358 0.0207922 0 - 0.0803724 2024-01-11T10:35:22.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1077 16 374 358 0.0207729 0 - 0.0803724 2024-01-11T10:35:23.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1078 16 374 358 0.0207536 0 - 0.0803724 2024-01-11T10:35:24.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1079 16 374 358 0.0207344 0 - 0.0803724 2024-01-11T10:35:25.230 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:35:25.255211+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:35:25.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:35:25.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1080 16 374 358 0.0207152 0 - 0.0803724 2024-01-11T10:35:26.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1081 16 374 358 0.020696 0 - 0.0803724 2024-01-11T10:35:27.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1082 16 374 358 0.0206769 0 - 0.0803724 2024-01-11T10:35:28.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1083 16 374 358 0.0206578 0 - 0.0803724 2024-01-11T10:35:29.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1084 16 374 358 0.0206387 0 - 0.0803724 2024-01-11T10:35:30.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1085 16 374 358 0.0206197 0 - 0.0803724 2024-01-11T10:35:31.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1086 16 374 358 0.0206007 0 - 0.0803724 2024-01-11T10:35:32.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1087 16 374 358 0.0205818 0 - 0.0803724 2024-01-11T10:35:33.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1088 16 374 358 0.0205629 0 - 0.0803724 2024-01-11T10:35:34.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1089 16 374 358 0.020544 0 - 0.0803724 2024-01-11T10:35:35.231 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1090 16 374 358 0.0205251 0 - 0.0803724 2024-01-11T10:35:36.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1091 16 374 358 0.0205063 0 - 0.0803724 2024-01-11T10:35:37.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1092 16 374 358 0.0204875 0 - 0.0803724 2024-01-11T10:35:38.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1093 16 374 358 0.0204688 0 - 0.0803724 2024-01-11T10:35:39.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1094 16 374 358 0.0204501 0 - 0.0803724 2024-01-11T10:35:40.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1095 16 374 358 0.0204314 0 - 0.0803724 2024-01-11T10:35:41.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1096 16 374 358 0.0204128 0 - 0.0803724 2024-01-11T10:35:42.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1097 16 374 358 0.0203942 0 - 0.0803724 2024-01-11T10:35:43.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1098 16 374 358 0.0203756 0 - 0.0803724 2024-01-11T10:35:43.437 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:35:43.437 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:35:44.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1099 16 374 358 0.020357 0 - 0.0803724 2024-01-11T10:35:45.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:35:45.257440+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:35:45.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:35:45.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1100 16 374 358 0.0203385 0 - 0.0803724 2024-01-11T10:35:45.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:35:45.556 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:35:45.579+0000 7fd67519e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:35:46.232 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1101 16 374 358 0.0203201 0 - 0.0803724 2024-01-11T10:35:47.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1102 16 374 358 0.0203016 0 - 0.0803724 2024-01-11T10:35:48.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1103 16 374 358 0.0202832 0 - 0.0803724 2024-01-11T10:35:49.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1104 16 374 358 0.0202649 0 - 0.0803724 2024-01-11T10:35:50.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1105 16 374 358 0.0202465 0 - 0.0803724 2024-01-11T10:35:51.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1106 16 374 358 0.0202282 0 - 0.0803724 2024-01-11T10:35:52.233 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1107 16 374 358 0.0202099 0 - 0.0803724 2024-01-11T10:35:53.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1108 16 374 358 0.0201917 0 - 0.0803724 2024-01-11T10:35:54.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1109 16 374 358 0.0201735 0 - 0.0803724 2024-01-11T10:35:55.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1110 16 374 358 0.0201553 0 - 0.0803724 2024-01-11T10:35:56.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1111 16 374 358 0.0201372 0 - 0.0803724 2024-01-11T10:35:57.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1112 16 374 358 0.0201191 0 - 0.0803724 2024-01-11T10:35:58.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1113 16 374 358 0.020101 0 - 0.0803724 2024-01-11T10:35:59.234 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1114 16 374 358 0.0200829 0 - 0.0803724 2024-01-11T10:36:00.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1115 16 374 358 0.0200649 0 - 0.0803724 2024-01-11T10:36:01.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1116 16 374 358 0.020047 0 - 0.0803724 2024-01-11T10:36:02.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1117 16 374 358 0.020029 0 - 0.0803724 2024-01-11T10:36:03.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1118 16 374 358 0.0200111 0 - 0.0803724 2024-01-11T10:36:04.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1119 16 374 358 0.0199932 0 - 0.0803724 2024-01-11T10:36:05.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:36:05.259417+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:36:05.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:36:05.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1120 16 374 358 0.0199754 0 - 0.0803724 2024-01-11T10:36:06.235 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1121 16 374 358 0.0199575 0 - 0.0803724 2024-01-11T10:36:07.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1122 16 374 358 0.0199398 0 - 0.0803724 2024-01-11T10:36:08.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1123 16 374 358 0.019922 0 - 0.0803724 2024-01-11T10:36:09.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1124 16 374 358 0.0199043 0 - 0.0803724 2024-01-11T10:36:10.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1125 16 374 358 0.0198866 0 - 0.0803724 2024-01-11T10:36:11.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1126 16 374 358 0.0198689 0 - 0.0803724 2024-01-11T10:36:12.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1127 16 374 358 0.0198513 0 - 0.0803724 2024-01-11T10:36:13.236 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1128 16 374 358 0.0198337 0 - 0.0803724 2024-01-11T10:36:14.237 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1129 16 374 358 0.0198161 0 - 0.0803724 2024-01-11T10:36:15.237 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1130 16 374 358 0.0197986 0 - 0.0803724 2024-01-11T10:36:16.237 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1131 16 374 358 0.0197811 0 - 0.0803724 2024-01-11T10:36:17.237 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1132 16 374 358 0.0197636 0 - 0.0803724 2024-01-11T10:36:18.237 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1133 16 374 358 0.0197462 0 - 0.0803724 2024-01-11T10:36:19.237 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1134 16 374 358 0.0197288 0 - 0.0803724 2024-01-11T10:36:20.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1135 16 374 358 0.0197114 0 - 0.0803724 2024-01-11T10:36:21.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1136 16 374 358 0.019694 0 - 0.0803724 2024-01-11T10:36:22.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1137 16 374 358 0.0196767 0 - 0.0803724 2024-01-11T10:36:23.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1138 16 374 358 0.0196594 0 - 0.0803724 2024-01-11T10:36:24.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1139 16 374 358 0.0196421 0 - 0.0803724 2024-01-11T10:36:25.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:36:25.261478+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:36:25.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:36:25.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1140 16 374 358 0.0196249 0 - 0.0803724 2024-01-11T10:36:26.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1141 16 374 358 0.0196077 0 - 0.0803724 2024-01-11T10:36:27.238 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1142 16 374 358 0.0195905 0 - 0.0803724 2024-01-11T10:36:28.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1143 16 374 358 0.0195734 0 - 0.0803724 2024-01-11T10:36:29.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1144 16 374 358 0.0195563 0 - 0.0803724 2024-01-11T10:36:30.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1145 16 374 358 0.0195392 0 - 0.0803724 2024-01-11T10:36:31.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1146 16 374 358 0.0195222 0 - 0.0803724 2024-01-11T10:36:32.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1147 16 374 358 0.0195052 0 - 0.0803724 2024-01-11T10:36:33.239 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1148 16 374 358 0.0194882 0 - 0.0803724 2024-01-11T10:36:34.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1149 16 374 358 0.0194712 0 - 0.0803724 2024-01-11T10:36:35.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1150 16 374 358 0.0194543 0 - 0.0803724 2024-01-11T10:36:36.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1151 16 374 358 0.0194374 0 - 0.0803724 2024-01-11T10:36:37.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1152 16 374 358 0.0194205 0 - 0.0803724 2024-01-11T10:36:38.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1153 16 374 358 0.0194036 0 - 0.0803724 2024-01-11T10:36:39.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1154 16 374 358 0.0193868 0 - 0.0803724 2024-01-11T10:36:40.240 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1155 16 374 358 0.01937 0 - 0.0803724 2024-01-11T10:36:41.241 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1156 16 374 358 0.0193533 0 - 0.0803724 2024-01-11T10:36:42.241 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1157 16 374 358 0.0193366 0 - 0.0803724 2024-01-11T10:36:43.241 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1158 16 374 358 0.0193199 0 - 0.0803724 2024-01-11T10:36:44.241 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1159 16 374 358 0.0193032 0 - 0.0803724 2024-01-11T10:36:45.241 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:36:45.263714+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:36:45.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:36:45.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1160 16 374 358 0.0192866 0 - 0.0803724 2024-01-11T10:36:46.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1161 16 374 358 0.0192699 0 - 0.0803724 2024-01-11T10:36:47.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1162 16 374 358 0.0192534 0 - 0.0803724 2024-01-11T10:36:48.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1163 16 374 358 0.0192368 0 - 0.0803724 2024-01-11T10:36:49.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1164 16 374 358 0.0192203 0 - 0.0803724 2024-01-11T10:36:50.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1165 16 374 358 0.0192038 0 - 0.0803724 2024-01-11T10:36:51.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1166 16 374 358 0.0191873 0 - 0.0803724 2024-01-11T10:36:52.242 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1167 16 374 358 0.0191709 0 - 0.0803724 2024-01-11T10:36:53.243 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1168 16 374 358 0.0191545 0 - 0.0803724 2024-01-11T10:36:54.243 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1169 16 374 358 0.0191381 0 - 0.0803724 2024-01-11T10:36:55.243 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1170 16 374 358 0.0191217 0 - 0.0803724 2024-01-11T10:36:56.243 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1171 16 374 358 0.0191054 0 - 0.0803724 2024-01-11T10:36:57.243 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1172 16 374 358 0.0190891 0 - 0.0803724 2024-01-11T10:36:58.243 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1173 16 374 358 0.0190728 0 - 0.0803724 2024-01-11T10:36:59.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1174 16 374 358 0.0190566 0 - 0.0803724 2024-01-11T10:37:00.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1175 16 374 358 0.0190403 0 - 0.0803724 2024-01-11T10:37:01.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1176 16 374 358 0.0190242 0 - 0.0803724 2024-01-11T10:37:02.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1177 16 374 358 0.019008 0 - 0.0803724 2024-01-11T10:37:03.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1178 16 374 358 0.0189919 0 - 0.0803724 2024-01-11T10:37:04.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1179 16 374 358 0.0189757 0 - 0.0803724 2024-01-11T10:37:05.244 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:37:05.265950+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:37:05.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:37:05.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1180 16 374 358 0.0189597 0 - 0.0803724 2024-01-11T10:37:06.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1181 16 374 358 0.0189436 0 - 0.0803724 2024-01-11T10:37:07.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1182 16 374 358 0.0189276 0 - 0.0803724 2024-01-11T10:37:08.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1183 16 374 358 0.0189116 0 - 0.0803724 2024-01-11T10:37:09.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1184 16 374 358 0.0188956 0 - 0.0803724 2024-01-11T10:37:10.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1185 16 374 358 0.0188797 0 - 0.0803724 2024-01-11T10:37:11.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1186 16 374 358 0.0188638 0 - 0.0803724 2024-01-11T10:37:12.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1187 16 374 358 0.0188479 0 - 0.0803724 2024-01-11T10:37:13.245 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1188 16 374 358 0.018832 0 - 0.0803724 2024-01-11T10:37:14.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1189 16 374 358 0.0188162 0 - 0.0803724 2024-01-11T10:37:15.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1190 16 374 358 0.0188003 0 - 0.0803724 2024-01-11T10:37:16.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1191 16 374 358 0.0187846 0 - 0.0803724 2024-01-11T10:37:17.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1192 16 374 358 0.0187688 0 - 0.0803724 2024-01-11T10:37:18.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1193 16 374 358 0.0187531 0 - 0.0803724 2024-01-11T10:37:19.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1194 16 374 358 0.0187374 0 - 0.0803724 2024-01-11T10:37:20.246 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1195 16 374 358 0.0187217 0 - 0.0803724 2024-01-11T10:37:21.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1196 16 374 358 0.018706 0 - 0.0803724 2024-01-11T10:37:22.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1197 16 374 358 0.0186904 0 - 0.0803724 2024-01-11T10:37:23.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1198 16 374 358 0.0186748 0 - 0.0803724 2024-01-11T10:37:24.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1199 16 374 358 0.0186592 0 - 0.0803724 2024-01-11T10:37:25.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:37:25.267979+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:37:25.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:37:25.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1200 16 374 358 0.0186437 0 - 0.0803724 2024-01-11T10:37:26.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1201 16 374 358 0.0186282 0 - 0.0803724 2024-01-11T10:37:27.247 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1202 16 374 358 0.0186127 0 - 0.0803724 2024-01-11T10:37:28.248 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1203 16 374 358 0.0185972 0 - 0.0803724 2024-01-11T10:37:29.248 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1204 16 374 358 0.0185817 0 - 0.0803724 2024-01-11T10:37:30.248 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1205 16 374 358 0.0185663 0 - 0.0803724 2024-01-11T10:37:31.248 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1206 16 374 358 0.0185509 0 - 0.0803724 2024-01-11T10:37:32.248 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1207 16 374 358 0.0185356 0 - 0.0803724 2024-01-11T10:37:33.248 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1208 16 374 358 0.0185202 0 - 0.0803724 2024-01-11T10:37:34.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1209 16 374 358 0.0185049 0 - 0.0803724 2024-01-11T10:37:35.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1210 16 374 358 0.0184896 0 - 0.0803724 2024-01-11T10:37:36.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1211 16 374 358 0.0184743 0 - 0.0803724 2024-01-11T10:37:37.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1212 16 374 358 0.0184591 0 - 0.0803724 2024-01-11T10:37:38.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1213 16 374 358 0.0184439 0 - 0.0803724 2024-01-11T10:37:39.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1214 16 374 358 0.0184287 0 - 0.0803724 2024-01-11T10:37:40.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1215 16 374 358 0.0184135 0 - 0.0803724 2024-01-11T10:37:41.249 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1216 16 374 358 0.0183984 0 - 0.0803724 2024-01-11T10:37:42.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1217 16 374 358 0.0183833 0 - 0.0803724 2024-01-11T10:37:43.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1218 16 374 358 0.0183682 0 - 0.0803724 2024-01-11T10:37:44.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1219 16 374 358 0.0183531 0 - 0.0803724 2024-01-11T10:37:45.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:37:45.270090+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:37:45.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:37:45.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1220 16 374 358 0.018338 0 - 0.0803724 2024-01-11T10:37:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:37:45.481 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:37:46.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1221 16 374 358 0.018323 0 - 0.0803724 2024-01-11T10:37:47.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1222 16 374 358 0.018308 0 - 0.0803724 2024-01-11T10:37:47.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:37:47.600 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:37:47.617+0000 7f5056d55640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:37:48.250 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1223 16 374 358 0.0182931 0 - 0.0803724 2024-01-11T10:37:49.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1224 16 374 358 0.0182781 0 - 0.0803724 2024-01-11T10:37:50.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1225 16 374 358 0.0182632 0 - 0.0803724 2024-01-11T10:37:51.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1226 16 374 358 0.0182483 0 - 0.0803724 2024-01-11T10:37:52.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1227 16 374 358 0.0182334 0 - 0.0803724 2024-01-11T10:37:53.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1228 16 374 358 0.0182186 0 - 0.0803724 2024-01-11T10:37:54.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1229 16 374 358 0.0182038 0 - 0.0803724 2024-01-11T10:37:55.251 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1230 16 374 358 0.018189 0 - 0.0803724 2024-01-11T10:37:56.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1231 16 374 358 0.0181742 0 - 0.0803724 2024-01-11T10:37:57.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1232 16 374 358 0.0181594 0 - 0.0803724 2024-01-11T10:37:58.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1233 16 374 358 0.0181447 0 - 0.0803724 2024-01-11T10:37:59.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1234 16 374 358 0.01813 0 - 0.0803724 2024-01-11T10:38:00.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1235 16 374 358 0.0181153 0 - 0.0803724 2024-01-11T10:38:01.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1236 16 374 358 0.0181007 0 - 0.0803724 2024-01-11T10:38:02.252 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1237 16 374 358 0.018086 0 - 0.0803724 2024-01-11T10:38:03.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1238 16 374 358 0.0180714 0 - 0.0803724 2024-01-11T10:38:04.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1239 16 374 358 0.0180568 0 - 0.0803724 2024-01-11T10:38:05.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:38:05.272072+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:38:05.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:38:05.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1240 16 374 358 0.0180423 0 - 0.0803724 2024-01-11T10:38:06.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1241 16 374 358 0.0180277 0 - 0.0803724 2024-01-11T10:38:07.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1242 16 374 358 0.0180132 0 - 0.0803724 2024-01-11T10:38:08.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1243 16 374 358 0.0179987 0 - 0.0803724 2024-01-11T10:38:09.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1244 16 374 358 0.0179843 0 - 0.0803724 2024-01-11T10:38:10.253 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1245 16 374 358 0.0179698 0 - 0.0803724 2024-01-11T10:38:11.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1246 16 374 358 0.0179554 0 - 0.0803724 2024-01-11T10:38:12.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1247 16 374 358 0.017941 0 - 0.0803724 2024-01-11T10:38:13.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1248 16 374 358 0.0179266 0 - 0.0803724 2024-01-11T10:38:14.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1249 16 374 358 0.0179123 0 - 0.0803724 2024-01-11T10:38:15.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1250 16 374 358 0.0178979 0 - 0.0803724 2024-01-11T10:38:16.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1251 16 374 358 0.0178836 0 - 0.0803724 2024-01-11T10:38:17.254 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1252 16 374 358 0.0178694 0 - 0.0803724 2024-01-11T10:38:18.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1253 16 374 358 0.0178551 0 - 0.0803724 2024-01-11T10:38:19.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1254 16 374 358 0.0178409 0 - 0.0803724 2024-01-11T10:38:20.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1255 16 374 358 0.0178266 0 - 0.0803724 2024-01-11T10:38:21.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1256 16 374 358 0.0178124 0 - 0.0803724 2024-01-11T10:38:22.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1257 16 374 358 0.0177983 0 - 0.0803724 2024-01-11T10:38:23.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1258 16 374 358 0.0177841 0 - 0.0803724 2024-01-11T10:38:24.255 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1259 16 374 358 0.01777 0 - 0.0803724 2024-01-11T10:38:25.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:38:25.274120+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:38:25.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:38:25.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1260 16 374 358 0.0177559 0 - 0.0803724 2024-01-11T10:38:26.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1261 16 374 358 0.0177418 0 - 0.0803724 2024-01-11T10:38:27.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1262 16 374 358 0.0177278 0 - 0.0803724 2024-01-11T10:38:28.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1263 16 374 358 0.0177137 0 - 0.0803724 2024-01-11T10:38:29.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1264 16 374 358 0.0176997 0 - 0.0803724 2024-01-11T10:38:30.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1265 16 374 358 0.0176857 0 - 0.0803724 2024-01-11T10:38:31.256 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1266 16 374 358 0.0176717 0 - 0.0803724 2024-01-11T10:38:32.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1267 16 374 358 0.0176578 0 - 0.0803724 2024-01-11T10:38:33.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1268 16 374 358 0.0176439 0 - 0.0803724 2024-01-11T10:38:34.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1269 16 374 358 0.01763 0 - 0.0803724 2024-01-11T10:38:35.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1270 16 374 358 0.0176161 0 - 0.0803724 2024-01-11T10:38:36.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1271 16 374 358 0.0176022 0 - 0.0803724 2024-01-11T10:38:37.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1272 16 374 358 0.0175884 0 - 0.0803724 2024-01-11T10:38:38.257 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1273 16 374 358 0.0175746 0 - 0.0803724 2024-01-11T10:38:39.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1274 16 374 358 0.0175608 0 - 0.0803724 2024-01-11T10:38:40.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1275 16 374 358 0.017547 0 - 0.0803724 2024-01-11T10:38:41.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1276 16 374 358 0.0175333 0 - 0.0803724 2024-01-11T10:38:42.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1277 16 374 358 0.0175195 0 - 0.0803724 2024-01-11T10:38:43.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1278 16 374 358 0.0175058 0 - 0.0803724 2024-01-11T10:38:44.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1279 16 374 358 0.0174921 0 - 0.0803724 2024-01-11T10:38:45.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:38:45.276304+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:38:45.258 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:38:45.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1280 16 374 358 0.0174785 0 - 0.0803724 2024-01-11T10:38:46.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1281 16 374 358 0.0174648 0 - 0.0803724 2024-01-11T10:38:47.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1282 16 374 358 0.0174512 0 - 0.0803724 2024-01-11T10:38:48.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1283 16 374 358 0.0174376 0 - 0.0803724 2024-01-11T10:38:49.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1284 16 374 358 0.017424 0 - 0.0803724 2024-01-11T10:38:50.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1285 16 374 358 0.0174105 0 - 0.0803724 2024-01-11T10:38:51.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1286 16 374 358 0.0173969 0 - 0.0803724 2024-01-11T10:38:52.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1287 16 374 358 0.0173834 0 - 0.0803724 2024-01-11T10:38:53.259 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1288 16 374 358 0.0173699 0 - 0.0803724 2024-01-11T10:38:54.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1289 16 374 358 0.0173564 0 - 0.0803724 2024-01-11T10:38:55.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1290 16 374 358 0.017343 0 - 0.0803724 2024-01-11T10:38:56.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1291 16 374 358 0.0173295 0 - 0.0803724 2024-01-11T10:38:57.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1292 16 374 358 0.0173161 0 - 0.0803724 2024-01-11T10:38:58.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1293 16 374 358 0.0173027 0 - 0.0803724 2024-01-11T10:38:59.260 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1294 16 374 358 0.0172894 0 - 0.0803724 2024-01-11T10:39:00.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1295 16 374 358 0.017276 0 - 0.0803724 2024-01-11T10:39:01.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1296 16 374 358 0.0172627 0 - 0.0803724 2024-01-11T10:39:02.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1297 16 374 358 0.0172494 0 - 0.0803724 2024-01-11T10:39:03.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1298 16 374 358 0.0172361 0 - 0.0803724 2024-01-11T10:39:04.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1299 16 374 358 0.0172228 0 - 0.0803724 2024-01-11T10:39:05.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:39:05.278515+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:39:05.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:39:05.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1300 16 374 358 0.0172096 0 - 0.0803724 2024-01-11T10:39:06.261 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1301 16 374 358 0.0171963 0 - 0.0803724 2024-01-11T10:39:07.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1302 16 374 358 0.0171831 0 - 0.0803724 2024-01-11T10:39:08.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1303 16 374 358 0.0171699 0 - 0.0803724 2024-01-11T10:39:09.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1304 16 374 358 0.0171568 0 - 0.0803724 2024-01-11T10:39:10.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1305 16 374 358 0.0171436 0 - 0.0803724 2024-01-11T10:39:11.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1306 16 374 358 0.0171305 0 - 0.0803724 2024-01-11T10:39:12.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1307 16 374 358 0.0171174 0 - 0.0803724 2024-01-11T10:39:13.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1308 16 374 358 0.0171043 0 - 0.0803724 2024-01-11T10:39:14.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1309 16 374 358 0.0170912 0 - 0.0803724 2024-01-11T10:39:15.262 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1310 16 374 358 0.0170782 0 - 0.0803724 2024-01-11T10:39:16.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1311 16 374 358 0.0170652 0 - 0.0803724 2024-01-11T10:39:17.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1312 16 374 358 0.0170522 0 - 0.0803724 2024-01-11T10:39:18.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1313 16 374 358 0.0170392 0 - 0.0803724 2024-01-11T10:39:19.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1314 16 374 358 0.0170262 0 - 0.0803724 2024-01-11T10:39:20.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1315 16 374 358 0.0170133 0 - 0.0803724 2024-01-11T10:39:21.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1316 16 374 358 0.0170003 0 - 0.0803724 2024-01-11T10:39:22.263 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1317 16 374 358 0.0169874 0 - 0.0803724 2024-01-11T10:39:23.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1318 16 374 358 0.0169745 0 - 0.0803724 2024-01-11T10:39:24.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1319 16 374 358 0.0169617 0 - 0.0803724 2024-01-11T10:39:25.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:39:25.280553+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:39:25.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:39:25.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1320 16 374 358 0.0169488 0 - 0.0803724 2024-01-11T10:39:26.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1321 16 374 358 0.016936 0 - 0.0803724 2024-01-11T10:39:27.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1322 16 374 358 0.0169232 0 - 0.0803724 2024-01-11T10:39:28.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1323 16 374 358 0.0169104 0 - 0.0803724 2024-01-11T10:39:29.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1324 16 374 358 0.0168976 0 - 0.0803724 2024-01-11T10:39:30.264 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1325 16 374 358 0.0168849 0 - 0.0803724 2024-01-11T10:39:31.265 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1326 16 374 358 0.0168721 0 - 0.0803724 2024-01-11T10:39:32.265 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1327 16 374 358 0.0168594 0 - 0.0803724 2024-01-11T10:39:33.265 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1328 16 374 358 0.0168467 0 - 0.0803724 2024-01-11T10:39:34.265 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1329 16 374 358 0.016834 0 - 0.0803724 2024-01-11T10:39:35.265 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1330 16 374 358 0.0168214 0 - 0.0803724 2024-01-11T10:39:36.265 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1331 16 374 358 0.0168087 0 - 0.0803724 2024-01-11T10:39:37.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1332 16 374 358 0.0167961 0 - 0.0803724 2024-01-11T10:39:38.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1333 16 374 358 0.0167835 0 - 0.0803724 2024-01-11T10:39:39.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1334 16 374 358 0.0167709 0 - 0.0803724 2024-01-11T10:39:40.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1335 16 374 358 0.0167584 0 - 0.0803724 2024-01-11T10:39:41.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1336 16 374 358 0.0167458 0 - 0.0803724 2024-01-11T10:39:42.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1337 16 374 358 0.0167333 0 - 0.0803724 2024-01-11T10:39:43.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1338 16 374 358 0.0167208 0 - 0.0803724 2024-01-11T10:39:44.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1339 16 374 358 0.0167083 0 - 0.0803724 2024-01-11T10:39:45.266 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:39:45.282588+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:39:45.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:39:45.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1340 16 374 358 0.0166959 0 - 0.0803724 2024-01-11T10:39:46.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1341 16 374 358 0.0166834 0 - 0.0803724 2024-01-11T10:39:47.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1342 16 374 358 0.016671 0 - 0.0803724 2024-01-11T10:39:47.524 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:39:47.524 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:39:48.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1343 16 374 358 0.0166586 0 - 0.0803724 2024-01-11T10:39:49.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1344 16 374 358 0.0166462 0 - 0.0803724 2024-01-11T10:39:49.526 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:39:49.645 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:39:49.658+0000 7f857175e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:39:50.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1345 16 374 358 0.0166338 0 - 0.0803724 2024-01-11T10:39:51.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1346 16 374 358 0.0166214 0 - 0.0803724 2024-01-11T10:39:52.267 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1347 16 374 358 0.0166091 0 - 0.0803724 2024-01-11T10:39:53.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1348 16 374 358 0.0165968 0 - 0.0803724 2024-01-11T10:39:54.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1349 16 374 358 0.0165845 0 - 0.0803724 2024-01-11T10:39:55.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1350 16 374 358 0.0165722 0 - 0.0803724 2024-01-11T10:39:56.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1351 16 374 358 0.0165599 0 - 0.0803724 2024-01-11T10:39:57.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1352 16 374 358 0.0165477 0 - 0.0803724 2024-01-11T10:39:58.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1353 16 374 358 0.0165354 0 - 0.0803724 2024-01-11T10:39:59.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1354 16 374 358 0.0165232 0 - 0.0803724 2024-01-11T10:40:00.268 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1355 16 374 358 0.016511 0 - 0.0803724 2024-01-11T10:40:01.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1356 16 374 358 0.0164989 0 - 0.0803724 2024-01-11T10:40:02.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1357 16 374 358 0.0164867 0 - 0.0803724 2024-01-11T10:40:03.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1358 16 374 358 0.0164746 0 - 0.0803724 2024-01-11T10:40:04.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1359 16 374 358 0.0164624 0 - 0.0803724 2024-01-11T10:40:05.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:40:05.284645+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:40:05.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:40:05.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1360 16 374 358 0.0164503 0 - 0.0803724 2024-01-11T10:40:06.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1361 16 374 358 0.0164382 0 - 0.0803724 2024-01-11T10:40:07.269 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1362 16 374 358 0.0164262 0 - 0.0803724 2024-01-11T10:40:08.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1363 16 374 358 0.0164141 0 - 0.0803724 2024-01-11T10:40:09.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1364 16 374 358 0.0164021 0 - 0.0803724 2024-01-11T10:40:10.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1365 16 374 358 0.0163901 0 - 0.0803724 2024-01-11T10:40:11.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1366 16 374 358 0.0163781 0 - 0.0803724 2024-01-11T10:40:12.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1367 16 374 358 0.0163661 0 - 0.0803724 2024-01-11T10:40:13.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1368 16 374 358 0.0163541 0 - 0.0803724 2024-01-11T10:40:14.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1369 16 374 358 0.0163422 0 - 0.0803724 2024-01-11T10:40:15.270 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1370 16 374 358 0.0163303 0 - 0.0803724 2024-01-11T10:40:16.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1371 16 374 358 0.0163183 0 - 0.0803724 2024-01-11T10:40:17.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1372 16 374 358 0.0163064 0 - 0.0803724 2024-01-11T10:40:18.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1373 16 374 358 0.0162946 0 - 0.0803724 2024-01-11T10:40:19.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1374 16 374 358 0.0162827 0 - 0.0803724 2024-01-11T10:40:20.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1375 16 374 358 0.0162709 0 - 0.0803724 2024-01-11T10:40:21.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1376 16 374 358 0.016259 0 - 0.0803724 2024-01-11T10:40:22.271 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1377 16 374 358 0.0162472 0 - 0.0803724 2024-01-11T10:40:23.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1378 16 374 358 0.0162354 0 - 0.0803724 2024-01-11T10:40:24.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1379 16 374 358 0.0162237 0 - 0.0803724 2024-01-11T10:40:25.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:40:25.286700+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:40:25.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:40:25.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1380 16 374 358 0.0162119 0 - 0.0803724 2024-01-11T10:40:26.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1381 16 374 358 0.0162002 0 - 0.0803724 2024-01-11T10:40:27.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1382 16 374 358 0.0161885 0 - 0.0803724 2024-01-11T10:40:28.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1383 16 374 358 0.0161768 0 - 0.0803724 2024-01-11T10:40:29.272 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1384 16 374 358 0.0161651 0 - 0.0803724 2024-01-11T10:40:30.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1385 16 374 358 0.0161534 0 - 0.0803724 2024-01-11T10:40:31.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1386 16 374 358 0.0161417 0 - 0.0803724 2024-01-11T10:40:32.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1387 16 374 358 0.0161301 0 - 0.0803724 2024-01-11T10:40:33.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1388 16 374 358 0.0161185 0 - 0.0803724 2024-01-11T10:40:34.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1389 16 374 358 0.0161069 0 - 0.0803724 2024-01-11T10:40:35.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1390 16 374 358 0.0160953 0 - 0.0803724 2024-01-11T10:40:36.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1391 16 374 358 0.0160837 0 - 0.0803724 2024-01-11T10:40:37.273 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1392 16 374 358 0.0160722 0 - 0.0803724 2024-01-11T10:40:38.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1393 16 374 358 0.0160606 0 - 0.0803724 2024-01-11T10:40:39.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1394 16 374 358 0.0160491 0 - 0.0803724 2024-01-11T10:40:40.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1395 16 374 358 0.0160376 0 - 0.0803724 2024-01-11T10:40:41.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1396 16 374 358 0.0160261 0 - 0.0803724 2024-01-11T10:40:42.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1397 16 374 358 0.0160146 0 - 0.0803724 2024-01-11T10:40:43.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1398 16 374 358 0.0160032 0 - 0.0803724 2024-01-11T10:40:44.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1399 16 374 358 0.0159917 0 - 0.0803724 2024-01-11T10:40:45.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:40:45.288819+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:40:45.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:40:45.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1400 16 374 358 0.0159803 0 - 0.0803724 2024-01-11T10:40:46.274 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1401 16 374 358 0.0159689 0 - 0.0803724 2024-01-11T10:40:47.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1402 16 374 358 0.0159575 0 - 0.0803724 2024-01-11T10:40:48.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1403 16 374 358 0.0159462 0 - 0.0803724 2024-01-11T10:40:49.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1404 16 374 358 0.0159348 0 - 0.0803724 2024-01-11T10:40:50.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1405 16 374 358 0.0159235 0 - 0.0803724 2024-01-11T10:40:51.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1406 16 374 358 0.0159121 0 - 0.0803724 2024-01-11T10:40:52.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1407 16 374 358 0.0159008 0 - 0.0803724 2024-01-11T10:40:53.275 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1408 16 374 358 0.0158895 0 - 0.0803724 2024-01-11T10:40:54.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1409 16 374 358 0.0158782 0 - 0.0803724 2024-01-11T10:40:55.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1410 16 374 358 0.015867 0 - 0.0803724 2024-01-11T10:40:56.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1411 16 374 358 0.0158557 0 - 0.0803724 2024-01-11T10:40:57.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1412 16 374 358 0.0158445 0 - 0.0803724 2024-01-11T10:40:58.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1413 16 374 358 0.0158333 0 - 0.0803724 2024-01-11T10:40:59.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1414 16 374 358 0.0158221 0 - 0.0803724 2024-01-11T10:41:00.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1415 16 374 358 0.0158109 0 - 0.0803724 2024-01-11T10:41:01.276 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1416 16 374 358 0.0157998 0 - 0.0803724 2024-01-11T10:41:02.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1417 16 374 358 0.0157886 0 - 0.0803724 2024-01-11T10:41:03.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1418 16 374 358 0.0157775 0 - 0.0803724 2024-01-11T10:41:04.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1419 16 374 358 0.0157664 0 - 0.0803724 2024-01-11T10:41:05.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:41:05.290807+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:41:05.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:41:05.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1420 16 374 358 0.0157553 0 - 0.0803724 2024-01-11T10:41:06.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1421 16 374 358 0.0157442 0 - 0.0803724 2024-01-11T10:41:07.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1422 16 374 358 0.0157331 0 - 0.0803724 2024-01-11T10:41:08.277 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1423 16 374 358 0.015722 0 - 0.0803724 2024-01-11T10:41:09.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1424 16 374 358 0.015711 0 - 0.0803724 2024-01-11T10:41:10.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1425 16 374 358 0.0157 0 - 0.0803724 2024-01-11T10:41:11.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1426 16 374 358 0.015689 0 - 0.0803724 2024-01-11T10:41:12.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1427 16 374 358 0.015678 0 - 0.0803724 2024-01-11T10:41:13.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1428 16 374 358 0.015667 0 - 0.0803724 2024-01-11T10:41:14.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1429 16 374 358 0.015656 0 - 0.0803724 2024-01-11T10:41:15.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1430 16 374 358 0.0156451 0 - 0.0803724 2024-01-11T10:41:16.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1431 16 374 358 0.0156341 0 - 0.0803724 2024-01-11T10:41:17.278 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1432 16 374 358 0.0156232 0 - 0.0803724 2024-01-11T10:41:18.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1433 16 374 358 0.0156123 0 - 0.0803724 2024-01-11T10:41:19.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1434 16 374 358 0.0156014 0 - 0.0803724 2024-01-11T10:41:20.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1435 16 374 358 0.0155906 0 - 0.0803724 2024-01-11T10:41:21.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1436 16 374 358 0.0155797 0 - 0.0803724 2024-01-11T10:41:22.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1437 16 374 358 0.0155689 0 - 0.0803724 2024-01-11T10:41:23.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1438 16 374 358 0.015558 0 - 0.0803724 2024-01-11T10:41:24.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1439 16 374 358 0.0155472 0 - 0.0803724 2024-01-11T10:41:25.279 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:41:25.292811+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:41:25.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:41:25.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1440 16 374 358 0.0155364 0 - 0.0803724 2024-01-11T10:41:26.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1441 16 374 358 0.0155256 0 - 0.0803724 2024-01-11T10:41:27.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1442 16 374 358 0.0155149 0 - 0.0803724 2024-01-11T10:41:28.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1443 16 374 358 0.0155041 0 - 0.0803724 2024-01-11T10:41:29.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1444 16 374 358 0.0154934 0 - 0.0803724 2024-01-11T10:41:30.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1445 16 374 358 0.0154827 0 - 0.0803724 2024-01-11T10:41:31.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1446 16 374 358 0.015472 0 - 0.0803724 2024-01-11T10:41:32.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1447 16 374 358 0.0154613 0 - 0.0803724 2024-01-11T10:41:33.280 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1448 16 374 358 0.0154506 0 - 0.0803724 2024-01-11T10:41:34.281 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1449 16 374 358 0.0154399 0 - 0.0803724 2024-01-11T10:41:35.281 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1450 16 374 358 0.0154293 0 - 0.0803724 2024-01-11T10:41:36.281 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1451 16 374 358 0.0154186 0 - 0.0803724 2024-01-11T10:41:37.281 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1452 16 374 358 0.015408 0 - 0.0803724 2024-01-11T10:41:38.281 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1453 16 374 358 0.0153974 0 - 0.0803724 2024-01-11T10:41:39.281 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1454 16 374 358 0.0153868 0 - 0.0803724 2024-01-11T10:41:40.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1455 16 374 358 0.0153763 0 - 0.0803724 2024-01-11T10:41:41.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1456 16 374 358 0.0153657 0 - 0.0803724 2024-01-11T10:41:42.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1457 16 374 358 0.0153552 0 - 0.0803724 2024-01-11T10:41:43.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1458 16 374 358 0.0153446 0 - 0.0803724 2024-01-11T10:41:44.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1459 16 374 358 0.0153341 0 - 0.0803724 2024-01-11T10:41:45.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:41:45.294979+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:41:45.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:41:45.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1460 16 374 358 0.0153236 0 - 0.0803724 2024-01-11T10:41:46.282 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1461 16 374 358 0.0153131 0 - 0.0803724 2024-01-11T10:41:47.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1462 16 374 358 0.0153026 0 - 0.0803724 2024-01-11T10:41:48.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1463 16 374 358 0.0152922 0 - 0.0803724 2024-01-11T10:41:49.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1464 16 374 358 0.0152817 0 - 0.0803724 2024-01-11T10:41:49.567 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:41:49.567 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:41:50.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1465 16 374 358 0.0152713 0 - 0.0803724 2024-01-11T10:41:51.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1466 16 374 358 0.0152609 0 - 0.0803724 2024-01-11T10:41:51.569 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:41:51.688 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:41:51.698+0000 7f8a4e39e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:41:52.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1467 16 374 358 0.0152505 0 - 0.0803724 2024-01-11T10:41:53.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1468 16 374 358 0.0152401 0 - 0.0803724 2024-01-11T10:41:54.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1469 16 374 358 0.0152297 0 - 0.0803724 2024-01-11T10:41:55.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1470 16 374 358 0.0152194 0 - 0.0803724 2024-01-11T10:41:56.283 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1471 16 374 358 0.015209 0 - 0.0803724 2024-01-11T10:41:57.284 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1472 16 374 358 0.0151987 0 - 0.0803724 2024-01-11T10:41:58.284 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1473 16 374 358 0.0151884 0 - 0.0803724 2024-01-11T10:41:59.284 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1474 16 374 358 0.0151781 0 - 0.0803724 2024-01-11T10:42:00.284 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1475 16 374 358 0.0151678 0 - 0.0803724 2024-01-11T10:42:01.284 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1476 16 374 358 0.0151575 0 - 0.0803724 2024-01-11T10:42:02.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1477 16 374 358 0.0151472 0 - 0.0803724 2024-01-11T10:42:03.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1478 16 374 358 0.015137 0 - 0.0803724 2024-01-11T10:42:04.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1479 16 374 358 0.0151268 0 - 0.0803724 2024-01-11T10:42:05.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:42:05.297085+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:42:05.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:42:05.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1480 16 374 358 0.0151165 0 - 0.0803724 2024-01-11T10:42:06.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1481 16 374 358 0.0151063 0 - 0.0803724 2024-01-11T10:42:07.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1482 16 374 358 0.0150961 0 - 0.0803724 2024-01-11T10:42:08.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1483 16 374 358 0.015086 0 - 0.0803724 2024-01-11T10:42:09.285 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1484 16 374 358 0.0150758 0 - 0.0803724 2024-01-11T10:42:10.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1485 16 374 358 0.0150656 0 - 0.0803724 2024-01-11T10:42:11.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1486 16 374 358 0.0150555 0 - 0.0803724 2024-01-11T10:42:12.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1487 16 374 358 0.0150454 0 - 0.0803724 2024-01-11T10:42:13.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1488 16 374 358 0.0150353 0 - 0.0803724 2024-01-11T10:42:14.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1489 16 374 358 0.0150252 0 - 0.0803724 2024-01-11T10:42:15.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1490 16 374 358 0.0150151 0 - 0.0803724 2024-01-11T10:42:16.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1491 16 374 358 0.015005 0 - 0.0803724 2024-01-11T10:42:17.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1492 16 374 358 0.014995 0 - 0.0803724 2024-01-11T10:42:18.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1493 16 374 358 0.0149849 0 - 0.0803724 2024-01-11T10:42:19.286 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1494 16 374 358 0.0149749 0 - 0.0803724 2024-01-11T10:42:20.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1495 16 374 358 0.0149649 0 - 0.0803724 2024-01-11T10:42:21.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1496 16 374 358 0.0149549 0 - 0.0803724 2024-01-11T10:42:22.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1497 16 374 358 0.0149449 0 - 0.0803724 2024-01-11T10:42:23.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1498 16 374 358 0.0149349 0 - 0.0803724 2024-01-11T10:42:24.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1499 16 374 358 0.0149249 0 - 0.0803724 2024-01-11T10:42:25.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:42:25.299024+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:42:25.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:42:25.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1500 16 374 358 0.014915 0 - 0.0803724 2024-01-11T10:42:26.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1501 16 374 358 0.014905 0 - 0.0803724 2024-01-11T10:42:27.287 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1502 16 374 358 0.0148951 0 - 0.0803724 2024-01-11T10:42:28.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1503 16 374 358 0.0148852 0 - 0.0803724 2024-01-11T10:42:29.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1504 16 374 358 0.0148753 0 - 0.0803724 2024-01-11T10:42:30.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1505 16 374 358 0.0148654 0 - 0.0803724 2024-01-11T10:42:31.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1506 16 374 358 0.0148556 0 - 0.0803724 2024-01-11T10:42:32.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1507 16 374 358 0.0148457 0 - 0.0803724 2024-01-11T10:42:33.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1508 16 374 358 0.0148359 0 - 0.0803724 2024-01-11T10:42:34.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1509 16 374 358 0.014826 0 - 0.0803724 2024-01-11T10:42:35.288 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1510 16 374 358 0.0148162 0 - 0.0803724 2024-01-11T10:42:36.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1511 16 374 358 0.0148064 0 - 0.0803724 2024-01-11T10:42:37.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1512 16 374 358 0.0147966 0 - 0.0803724 2024-01-11T10:42:38.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1513 16 374 358 0.0147868 0 - 0.0803724 2024-01-11T10:42:39.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1514 16 374 358 0.0147771 0 - 0.0803724 2024-01-11T10:42:40.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1515 16 374 358 0.0147673 0 - 0.0803724 2024-01-11T10:42:41.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1516 16 374 358 0.0147576 0 - 0.0803724 2024-01-11T10:42:42.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1517 16 374 358 0.0147478 0 - 0.0803724 2024-01-11T10:42:43.289 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1518 16 374 358 0.0147381 0 - 0.0803724 2024-01-11T10:42:44.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1519 16 374 358 0.0147284 0 - 0.0803724 2024-01-11T10:42:45.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:42:45.301107+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:42:45.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:42:45.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1520 16 374 358 0.0147187 0 - 0.0803724 2024-01-11T10:42:46.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1521 16 374 358 0.0147091 0 - 0.0803724 2024-01-11T10:42:47.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1522 16 374 358 0.0146994 0 - 0.0803724 2024-01-11T10:42:48.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1523 16 374 358 0.0146897 0 - 0.0803724 2024-01-11T10:42:49.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1524 16 374 358 0.0146801 0 - 0.0803724 2024-01-11T10:42:50.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1525 16 374 358 0.0146705 0 - 0.0803724 2024-01-11T10:42:51.290 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1526 16 374 358 0.0146609 0 - 0.0803724 2024-01-11T10:42:52.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1527 16 374 358 0.0146513 0 - 0.0803724 2024-01-11T10:42:53.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1528 16 374 358 0.0146417 0 - 0.0803724 2024-01-11T10:42:54.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1529 16 374 358 0.0146321 0 - 0.0803724 2024-01-11T10:42:55.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1530 16 374 358 0.0146225 0 - 0.0803724 2024-01-11T10:42:56.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1531 16 374 358 0.014613 0 - 0.0803724 2024-01-11T10:42:57.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1532 16 374 358 0.0146034 0 - 0.0803724 2024-01-11T10:42:58.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1533 16 374 358 0.0145939 0 - 0.0803724 2024-01-11T10:42:59.291 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1534 16 374 358 0.0145844 0 - 0.0803724 2024-01-11T10:43:00.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1535 16 374 358 0.0145749 0 - 0.0803724 2024-01-11T10:43:01.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1536 16 374 358 0.0145654 0 - 0.0803724 2024-01-11T10:43:02.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1537 16 374 358 0.0145559 0 - 0.0803724 2024-01-11T10:43:03.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1538 16 374 358 0.0145465 0 - 0.0803724 2024-01-11T10:43:04.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1539 16 374 358 0.014537 0 - 0.0803724 2024-01-11T10:43:05.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:43:05.303179+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:43:05.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:43:05.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1540 16 374 358 0.0145276 0 - 0.0803724 2024-01-11T10:43:06.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1541 16 374 358 0.0145182 0 - 0.0803724 2024-01-11T10:43:07.292 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1542 16 374 358 0.0145087 0 - 0.0803724 2024-01-11T10:43:08.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1543 16 374 358 0.0144993 0 - 0.0803724 2024-01-11T10:43:09.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1544 16 374 358 0.0144899 0 - 0.0803724 2024-01-11T10:43:10.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1545 16 374 358 0.0144806 0 - 0.0803724 2024-01-11T10:43:11.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1546 16 374 358 0.0144712 0 - 0.0803724 2024-01-11T10:43:12.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1547 16 374 358 0.0144618 0 - 0.0803724 2024-01-11T10:43:13.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1548 16 374 358 0.0144525 0 - 0.0803724 2024-01-11T10:43:14.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1549 16 374 358 0.0144432 0 - 0.0803724 2024-01-11T10:43:15.293 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1550 16 374 358 0.0144339 0 - 0.0803724 2024-01-11T10:43:16.294 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1551 16 374 358 0.0144245 0 - 0.0803724 2024-01-11T10:43:17.294 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1552 16 374 358 0.0144153 0 - 0.0803724 2024-01-11T10:43:18.294 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1553 16 374 358 0.014406 0 - 0.0803724 2024-01-11T10:43:19.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1554 16 374 358 0.0143967 0 - 0.0803724 2024-01-11T10:43:20.294 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1555 16 374 358 0.0143874 0 - 0.0803724 2024-01-11T10:43:21.294 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1556 16 374 358 0.0143782 0 - 0.0803724 2024-01-11T10:43:22.294 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1557 16 374 358 0.014369 0 - 0.0803724 2024-01-11T10:43:23.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1558 16 374 358 0.0143597 0 - 0.0803724 2024-01-11T10:43:24.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1559 16 374 358 0.0143505 0 - 0.0803724 2024-01-11T10:43:25.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:43:25.305310+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:43:25.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:43:25.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1560 16 374 358 0.0143413 0 - 0.0803724 2024-01-11T10:43:26.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1561 16 374 358 0.0143321 0 - 0.0803724 2024-01-11T10:43:27.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1562 16 374 358 0.014323 0 - 0.0803724 2024-01-11T10:43:28.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1563 16 374 358 0.0143138 0 - 0.0803724 2024-01-11T10:43:29.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1564 16 374 358 0.0143047 0 - 0.0803724 2024-01-11T10:43:30.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1565 16 374 358 0.0142955 0 - 0.0803724 2024-01-11T10:43:31.295 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1566 16 374 358 0.0142864 0 - 0.0803724 2024-01-11T10:43:32.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1567 16 374 358 0.0142773 0 - 0.0803724 2024-01-11T10:43:33.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1568 16 374 358 0.0142682 0 - 0.0803724 2024-01-11T10:43:34.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1569 16 374 358 0.0142591 0 - 0.0803724 2024-01-11T10:43:35.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1570 16 374 358 0.01425 0 - 0.0803724 2024-01-11T10:43:36.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1571 16 374 358 0.0142409 0 - 0.0803724 2024-01-11T10:43:37.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1572 16 374 358 0.0142319 0 - 0.0803724 2024-01-11T10:43:38.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1573 16 374 358 0.0142228 0 - 0.0803724 2024-01-11T10:43:39.296 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1574 16 374 358 0.0142138 0 - 0.0803724 2024-01-11T10:43:40.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1575 16 374 358 0.0142047 0 - 0.0803724 2024-01-11T10:43:41.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1576 16 374 358 0.0141957 0 - 0.0803724 2024-01-11T10:43:42.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1577 16 374 358 0.0141867 0 - 0.0803724 2024-01-11T10:43:43.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1578 16 374 358 0.0141777 0 - 0.0803724 2024-01-11T10:43:44.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1579 16 374 358 0.0141688 0 - 0.0803724 2024-01-11T10:43:45.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:43:45.307448+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:43:45.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:43:45.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1580 16 374 358 0.0141598 0 - 0.0803724 2024-01-11T10:43:46.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1581 16 374 358 0.0141508 0 - 0.0803724 2024-01-11T10:43:47.297 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1582 16 374 358 0.0141419 0 - 0.0803724 2024-01-11T10:43:48.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1583 16 374 358 0.014133 0 - 0.0803724 2024-01-11T10:43:49.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1584 16 374 358 0.014124 0 - 0.0803724 2024-01-11T10:43:50.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1585 16 374 358 0.0141151 0 - 0.0803724 2024-01-11T10:43:51.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1586 16 374 358 0.0141062 0 - 0.0803724 2024-01-11T10:43:51.610 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:43:51.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:43:52.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1587 16 374 358 0.0140973 0 - 0.0803724 2024-01-11T10:43:53.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1588 16 374 358 0.0140885 0 - 0.0803724 2024-01-11T10:43:53.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:43:53.729 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:43:53.736+0000 7ffbdbd08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:43:54.298 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1589 16 374 358 0.0140796 0 - 0.0803724 2024-01-11T10:43:55.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1590 16 374 358 0.0140707 0 - 0.0803724 2024-01-11T10:43:56.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1591 16 374 358 0.0140619 0 - 0.0803724 2024-01-11T10:43:57.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1592 16 374 358 0.0140531 0 - 0.0803724 2024-01-11T10:43:58.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1593 16 374 358 0.0140442 0 - 0.0803724 2024-01-11T10:43:59.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1594 16 374 358 0.0140354 0 - 0.0803724 2024-01-11T10:44:00.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1595 16 374 358 0.0140266 0 - 0.0803724 2024-01-11T10:44:01.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1596 16 374 358 0.0140178 0 - 0.0803724 2024-01-11T10:44:02.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1597 16 374 358 0.0140091 0 - 0.0803724 2024-01-11T10:44:03.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1598 16 374 358 0.0140003 0 - 0.0803724 2024-01-11T10:44:04.299 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1599 16 374 358 0.0139915 0 - 0.0803724 2024-01-11T10:44:05.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:44:05.309475+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:44:05.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:44:05.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1600 16 374 358 0.0139828 0 - 0.0803724 2024-01-11T10:44:06.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1601 16 374 358 0.0139741 0 - 0.0803724 2024-01-11T10:44:07.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1602 16 374 358 0.0139653 0 - 0.0803724 2024-01-11T10:44:08.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1603 16 374 358 0.0139566 0 - 0.0803724 2024-01-11T10:44:09.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1604 16 374 358 0.0139479 0 - 0.0803724 2024-01-11T10:44:10.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1605 16 374 358 0.0139392 0 - 0.0803724 2024-01-11T10:44:11.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1606 16 374 358 0.0139306 0 - 0.0803724 2024-01-11T10:44:12.300 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1607 16 374 358 0.0139219 0 - 0.0803724 2024-01-11T10:44:13.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1608 16 374 358 0.0139132 0 - 0.0803724 2024-01-11T10:44:14.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1609 16 374 358 0.0139046 0 - 0.0803724 2024-01-11T10:44:15.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1610 16 374 358 0.013896 0 - 0.0803724 2024-01-11T10:44:16.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1611 16 374 358 0.0138873 0 - 0.0803724 2024-01-11T10:44:17.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1612 16 374 358 0.0138787 0 - 0.0803724 2024-01-11T10:44:18.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1613 16 374 358 0.0138701 0 - 0.0803724 2024-01-11T10:44:19.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1614 16 374 358 0.0138615 0 - 0.0803724 2024-01-11T10:44:20.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1615 16 374 358 0.0138529 0 - 0.0803724 2024-01-11T10:44:21.301 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1616 16 374 358 0.0138444 0 - 0.0803724 2024-01-11T10:44:22.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1617 16 374 358 0.0138358 0 - 0.0803724 2024-01-11T10:44:23.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1618 16 374 358 0.0138272 0 - 0.0803724 2024-01-11T10:44:24.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1619 16 374 358 0.0138187 0 - 0.0803724 2024-01-11T10:44:25.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:44:25.311468+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:44:25.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:44:25.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1620 16 374 358 0.0138102 0 - 0.0803724 2024-01-11T10:44:26.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1621 16 374 358 0.0138017 0 - 0.0803724 2024-01-11T10:44:27.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1622 16 374 358 0.0137931 0 - 0.0803724 2024-01-11T10:44:28.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1623 16 374 358 0.0137846 0 - 0.0803724 2024-01-11T10:44:29.302 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1624 16 374 358 0.0137762 0 - 0.0803724 2024-01-11T10:44:30.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1625 16 374 358 0.0137677 0 - 0.0803724 2024-01-11T10:44:31.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1626 16 374 358 0.0137592 0 - 0.0803724 2024-01-11T10:44:32.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1627 16 374 358 0.0137508 0 - 0.0803724 2024-01-11T10:44:33.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1628 16 374 358 0.0137423 0 - 0.0803724 2024-01-11T10:44:34.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1629 16 374 358 0.0137339 0 - 0.0803724 2024-01-11T10:44:35.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1630 16 374 358 0.0137255 0 - 0.0803724 2024-01-11T10:44:36.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1631 16 374 358 0.013717 0 - 0.0803724 2024-01-11T10:44:37.303 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1632 16 374 358 0.0137086 0 - 0.0803724 2024-01-11T10:44:38.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1633 16 374 358 0.0137002 0 - 0.0803724 2024-01-11T10:44:39.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1634 16 374 358 0.0136919 0 - 0.0803724 2024-01-11T10:44:40.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1635 16 374 358 0.0136835 0 - 0.0803724 2024-01-11T10:44:41.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1636 16 374 358 0.0136751 0 - 0.0803724 2024-01-11T10:44:42.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1637 16 374 358 0.0136668 0 - 0.0803724 2024-01-11T10:44:43.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1638 16 374 358 0.0136584 0 - 0.0803724 2024-01-11T10:44:44.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1639 16 374 358 0.0136501 0 - 0.0803724 2024-01-11T10:44:45.304 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:44:45.313449+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:44:45.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:44:45.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1640 16 374 358 0.0136418 0 - 0.0803724 2024-01-11T10:44:46.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1641 16 374 358 0.0136334 0 - 0.0803724 2024-01-11T10:44:47.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1642 16 374 358 0.0136251 0 - 0.0803724 2024-01-11T10:44:48.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1643 16 374 358 0.0136169 0 - 0.0803724 2024-01-11T10:44:49.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1644 16 374 358 0.0136086 0 - 0.0803724 2024-01-11T10:44:50.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1645 16 374 358 0.0136003 0 - 0.0803724 2024-01-11T10:44:51.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1646 16 374 358 0.013592 0 - 0.0803724 2024-01-11T10:44:52.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1647 16 374 358 0.0135838 0 - 0.0803724 2024-01-11T10:44:53.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1648 16 374 358 0.0135755 0 - 0.0803724 2024-01-11T10:44:54.305 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1649 16 374 358 0.0135673 0 - 0.0803724 2024-01-11T10:44:55.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1650 16 374 358 0.0135591 0 - 0.0803724 2024-01-11T10:44:56.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1651 16 374 358 0.0135509 0 - 0.0803724 2024-01-11T10:44:57.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1652 16 374 358 0.0135427 0 - 0.0803724 2024-01-11T10:44:58.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1653 16 374 358 0.0135345 0 - 0.0803724 2024-01-11T10:44:59.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1654 16 374 358 0.0135263 0 - 0.0803724 2024-01-11T10:45:00.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1655 16 374 358 0.0135181 0 - 0.0803724 2024-01-11T10:45:01.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1656 16 374 358 0.01351 0 - 0.0803724 2024-01-11T10:45:02.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1657 16 374 358 0.0135018 0 - 0.0803724 2024-01-11T10:45:03.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1658 16 374 358 0.0134937 0 - 0.0803724 2024-01-11T10:45:04.306 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1659 16 374 358 0.0134855 0 - 0.0803724 2024-01-11T10:45:05.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:45:05.315397+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:45:05.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:45:05.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1660 16 374 358 0.0134774 0 - 0.0803724 2024-01-11T10:45:06.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1661 16 374 358 0.0134693 0 - 0.0803724 2024-01-11T10:45:07.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1662 16 374 358 0.0134612 0 - 0.0803724 2024-01-11T10:45:08.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1663 16 374 358 0.0134531 0 - 0.0803724 2024-01-11T10:45:09.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1664 16 374 358 0.013445 0 - 0.0803724 2024-01-11T10:45:10.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1665 16 374 358 0.0134369 0 - 0.0803724 2024-01-11T10:45:11.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1666 16 374 358 0.0134289 0 - 0.0803724 2024-01-11T10:45:12.307 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1667 16 374 358 0.0134208 0 - 0.0803724 2024-01-11T10:45:13.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1668 16 374 358 0.0134128 0 - 0.0803724 2024-01-11T10:45:14.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1669 16 374 358 0.0134047 0 - 0.0803724 2024-01-11T10:45:15.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1670 16 374 358 0.0133967 0 - 0.0803724 2024-01-11T10:45:16.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1671 16 374 358 0.0133887 0 - 0.0803724 2024-01-11T10:45:17.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1672 16 374 358 0.0133807 0 - 0.0803724 2024-01-11T10:45:18.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1673 16 374 358 0.0133727 0 - 0.0803724 2024-01-11T10:45:19.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1674 16 374 358 0.0133647 0 - 0.0803724 2024-01-11T10:45:20.308 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1675 16 374 358 0.0133567 0 - 0.0803724 2024-01-11T10:45:21.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1676 16 374 358 0.0133487 0 - 0.0803724 2024-01-11T10:45:22.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1677 16 374 358 0.0133408 0 - 0.0803724 2024-01-11T10:45:23.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1678 16 374 358 0.0133328 0 - 0.0803724 2024-01-11T10:45:24.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1679 16 374 358 0.0133249 0 - 0.0803724 2024-01-11T10:45:25.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:45:25.317402+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:45:25.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:45:25.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1680 16 374 358 0.013317 0 - 0.0803724 2024-01-11T10:45:26.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1681 16 374 358 0.013309 0 - 0.0803724 2024-01-11T10:45:27.309 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1682 16 374 358 0.0133011 0 - 0.0803724 2024-01-11T10:45:28.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1683 16 374 358 0.0132932 0 - 0.0803724 2024-01-11T10:45:29.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1684 16 374 358 0.0132853 0 - 0.0803724 2024-01-11T10:45:30.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1685 16 374 358 0.0132774 0 - 0.0803724 2024-01-11T10:45:31.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1686 16 374 358 0.0132696 0 - 0.0803724 2024-01-11T10:45:32.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1687 16 374 358 0.0132617 0 - 0.0803724 2024-01-11T10:45:33.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1688 16 374 358 0.0132538 0 - 0.0803724 2024-01-11T10:45:34.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1689 16 374 358 0.013246 0 - 0.0803724 2024-01-11T10:45:35.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1690 16 374 358 0.0132382 0 - 0.0803724 2024-01-11T10:45:36.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1691 16 374 358 0.0132303 0 - 0.0803724 2024-01-11T10:45:37.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1692 16 374 358 0.0132225 0 - 0.0803724 2024-01-11T10:45:38.310 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1693 16 374 358 0.0132147 0 - 0.0803724 2024-01-11T10:45:39.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1694 16 374 358 0.0132069 0 - 0.0803724 2024-01-11T10:45:40.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1695 16 374 358 0.0131991 0 - 0.0803724 2024-01-11T10:45:41.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1696 16 374 358 0.0131913 0 - 0.0803724 2024-01-11T10:45:42.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1697 16 374 358 0.0131836 0 - 0.0803724 2024-01-11T10:45:43.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1698 16 374 358 0.0131758 0 - 0.0803724 2024-01-11T10:45:44.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1699 16 374 358 0.013168 0 - 0.0803724 2024-01-11T10:45:45.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:45:45.319456+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:45:45.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:45:45.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1700 16 374 358 0.0131603 0 - 0.0803724 2024-01-11T10:45:46.311 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1701 16 374 358 0.0131526 0 - 0.0803724 2024-01-11T10:45:47.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1702 16 374 358 0.0131448 0 - 0.0803724 2024-01-11T10:45:48.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1703 16 374 358 0.0131371 0 - 0.0803724 2024-01-11T10:45:49.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1704 16 374 358 0.0131294 0 - 0.0803724 2024-01-11T10:45:50.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1705 16 374 358 0.0131217 0 - 0.0803724 2024-01-11T10:45:51.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1706 16 374 358 0.013114 0 - 0.0803724 2024-01-11T10:45:52.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1707 16 374 358 0.0131063 0 - 0.0803724 2024-01-11T10:45:53.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1708 16 374 358 0.0130987 0 - 0.0803724 2024-01-11T10:45:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:45:53.651 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:45:54.312 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1709 16 374 358 0.013091 0 - 0.0803724 2024-01-11T10:45:55.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1710 16 374 358 0.0130833 0 - 0.0803724 2024-01-11T10:45:55.652 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:45:55.759 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:45:55.764+0000 7fc2f0b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:45:56.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1711 16 374 358 0.0130757 0 - 0.0803724 2024-01-11T10:45:57.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1712 16 374 358 0.013068 0 - 0.0803724 2024-01-11T10:45:58.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1713 16 374 358 0.0130604 0 - 0.0803724 2024-01-11T10:45:59.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1714 16 374 358 0.0130528 0 - 0.0803724 2024-01-11T10:46:00.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1715 16 374 358 0.0130452 0 - 0.0803724 2024-01-11T10:46:01.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1716 16 374 358 0.0130376 0 - 0.0803724 2024-01-11T10:46:02.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1717 16 374 358 0.01303 0 - 0.0803724 2024-01-11T10:46:03.313 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1718 16 374 358 0.0130224 0 - 0.0803724 2024-01-11T10:46:04.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1719 16 374 358 0.0130148 0 - 0.0803724 2024-01-11T10:46:05.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:46:05.321488+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:46:05.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:46:05.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1720 16 374 358 0.0130073 0 - 0.0803724 2024-01-11T10:46:06.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1721 16 374 358 0.0129997 0 - 0.0803724 2024-01-11T10:46:07.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1722 16 374 358 0.0129922 0 - 0.0803724 2024-01-11T10:46:08.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1723 16 374 358 0.0129846 0 - 0.0803724 2024-01-11T10:46:09.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1724 16 374 358 0.0129771 0 - 0.0803724 2024-01-11T10:46:10.314 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1725 16 374 358 0.0129696 0 - 0.0803724 2024-01-11T10:46:11.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1726 16 374 358 0.0129621 0 - 0.0803724 2024-01-11T10:46:12.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1727 16 374 358 0.0129545 0 - 0.0803724 2024-01-11T10:46:13.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1728 16 374 358 0.012947 0 - 0.0803724 2024-01-11T10:46:14.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1729 16 374 358 0.0129396 0 - 0.0803724 2024-01-11T10:46:15.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1730 16 374 358 0.0129321 0 - 0.0803724 2024-01-11T10:46:16.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1731 16 374 358 0.0129246 0 - 0.0803724 2024-01-11T10:46:17.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1732 16 374 358 0.0129171 0 - 0.0803724 2024-01-11T10:46:18.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1733 16 374 358 0.0129097 0 - 0.0803724 2024-01-11T10:46:19.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1734 16 374 358 0.0129023 0 - 0.0803724 2024-01-11T10:46:20.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1735 16 374 358 0.0128948 0 - 0.0803724 2024-01-11T10:46:21.315 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1736 16 374 358 0.0128874 0 - 0.0803724 2024-01-11T10:46:22.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1737 16 374 358 0.01288 0 - 0.0803724 2024-01-11T10:46:23.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1738 16 374 358 0.0128726 0 - 0.0803724 2024-01-11T10:46:24.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1739 16 374 358 0.0128652 0 - 0.0803724 2024-01-11T10:46:25.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:46:25.323538+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:46:25.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:46:25.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1740 16 374 358 0.0128578 0 - 0.0803724 2024-01-11T10:46:26.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1741 16 374 358 0.0128504 0 - 0.0803724 2024-01-11T10:46:27.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1742 16 374 358 0.012843 0 - 0.0803724 2024-01-11T10:46:28.316 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1743 16 374 358 0.0128356 0 - 0.0803724 2024-01-11T10:46:29.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1744 16 374 358 0.0128283 0 - 0.0803724 2024-01-11T10:46:30.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1745 16 374 358 0.0128209 0 - 0.0803724 2024-01-11T10:46:31.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1746 16 374 358 0.0128136 0 - 0.0803724 2024-01-11T10:46:32.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1747 16 374 358 0.0128062 0 - 0.0803724 2024-01-11T10:46:33.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1748 16 374 358 0.0127989 0 - 0.0803724 2024-01-11T10:46:34.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1749 16 374 358 0.0127916 0 - 0.0803724 2024-01-11T10:46:35.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1750 16 374 358 0.0127843 0 - 0.0803724 2024-01-11T10:46:36.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1751 16 374 358 0.012777 0 - 0.0803724 2024-01-11T10:46:37.317 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1752 16 374 358 0.0127697 0 - 0.0803724 2024-01-11T10:46:38.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1753 16 374 358 0.0127624 0 - 0.0803724 2024-01-11T10:46:39.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1754 16 374 358 0.0127551 0 - 0.0803724 2024-01-11T10:46:40.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1755 16 374 358 0.0127479 0 - 0.0803724 2024-01-11T10:46:41.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1756 16 374 358 0.0127406 0 - 0.0803724 2024-01-11T10:46:42.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1757 16 374 358 0.0127334 0 - 0.0803724 2024-01-11T10:46:43.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1758 16 374 358 0.0127261 0 - 0.0803724 2024-01-11T10:46:44.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1759 16 374 358 0.0127189 0 - 0.0803724 2024-01-11T10:46:45.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:46:45.325592+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:46:45.318 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:46:45.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1760 16 374 358 0.0127117 0 - 0.0803724 2024-01-11T10:46:46.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1761 16 374 358 0.0127044 0 - 0.0803724 2024-01-11T10:46:47.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1762 16 374 358 0.0126972 0 - 0.0803724 2024-01-11T10:46:48.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1763 16 374 358 0.01269 0 - 0.0803724 2024-01-11T10:46:49.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1764 16 374 358 0.0126828 0 - 0.0803724 2024-01-11T10:46:50.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1765 16 374 358 0.0126756 0 - 0.0803724 2024-01-11T10:46:51.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1766 16 374 358 0.0126685 0 - 0.0803724 2024-01-11T10:46:52.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1767 16 374 358 0.0126613 0 - 0.0803724 2024-01-11T10:46:53.319 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1768 16 374 358 0.0126541 0 - 0.0803724 2024-01-11T10:46:54.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1769 16 374 358 0.012647 0 - 0.0803724 2024-01-11T10:46:55.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1770 16 374 358 0.0126398 0 - 0.0803724 2024-01-11T10:46:56.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1771 16 374 358 0.0126327 0 - 0.0803724 2024-01-11T10:46:57.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1772 16 374 358 0.0126256 0 - 0.0803724 2024-01-11T10:46:58.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1773 16 374 358 0.0126184 0 - 0.0803724 2024-01-11T10:46:59.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1774 16 374 358 0.0126113 0 - 0.0803724 2024-01-11T10:47:00.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1775 16 374 358 0.0126042 0 - 0.0803724 2024-01-11T10:47:01.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1776 16 374 358 0.0125971 0 - 0.0803724 2024-01-11T10:47:02.320 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1777 16 374 358 0.01259 0 - 0.0803724 2024-01-11T10:47:03.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1778 16 374 358 0.012583 0 - 0.0803724 2024-01-11T10:47:04.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1779 16 374 358 0.0125759 0 - 0.0803724 2024-01-11T10:47:05.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:47:05.327674+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:47:05.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:47:05.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1780 16 374 358 0.0125688 0 - 0.0803724 2024-01-11T10:47:06.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1781 16 374 358 0.0125618 0 - 0.0803724 2024-01-11T10:47:07.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1782 16 374 358 0.0125547 0 - 0.0803724 2024-01-11T10:47:08.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1783 16 374 358 0.0125477 0 - 0.0803724 2024-01-11T10:47:09.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1784 16 374 358 0.0125406 0 - 0.0803724 2024-01-11T10:47:10.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1785 16 374 358 0.0125336 0 - 0.0803724 2024-01-11T10:47:11.321 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1786 16 374 358 0.0125266 0 - 0.0803724 2024-01-11T10:47:12.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1787 16 374 358 0.0125196 0 - 0.0803724 2024-01-11T10:47:13.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1788 16 374 358 0.0125126 0 - 0.0803724 2024-01-11T10:47:14.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1789 16 374 358 0.0125056 0 - 0.0803724 2024-01-11T10:47:15.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1790 16 374 358 0.0124986 0 - 0.0803724 2024-01-11T10:47:16.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1791 16 374 358 0.0124916 0 - 0.0803724 2024-01-11T10:47:17.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1792 16 374 358 0.0124847 0 - 0.0803724 2024-01-11T10:47:18.322 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1793 16 374 358 0.0124777 0 - 0.0803724 2024-01-11T10:47:19.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1794 16 374 358 0.0124707 0 - 0.0803724 2024-01-11T10:47:20.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1795 16 374 358 0.0124638 0 - 0.0803724 2024-01-11T10:47:21.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1796 16 374 358 0.0124569 0 - 0.0803724 2024-01-11T10:47:22.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1797 16 374 358 0.0124499 0 - 0.0803724 2024-01-11T10:47:23.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1798 16 374 358 0.012443 0 - 0.0803724 2024-01-11T10:47:24.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1799 16 374 358 0.0124361 0 - 0.0803724 2024-01-11T10:47:25.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:47:25.329839+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:47:25.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:47:25.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1800 16 374 358 0.0124292 0 - 0.0803724 2024-01-11T10:47:26.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1801 16 374 358 0.0124223 0 - 0.0803724 2024-01-11T10:47:27.323 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1802 16 374 358 0.0124154 0 - 0.0803724 2024-01-11T10:47:28.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1803 16 374 358 0.0124085 0 - 0.0803724 2024-01-11T10:47:29.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1804 16 374 358 0.0124016 0 - 0.0803724 2024-01-11T10:47:30.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1805 16 374 358 0.0123947 0 - 0.0803724 2024-01-11T10:47:31.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1806 16 374 358 0.0123879 0 - 0.0803724 2024-01-11T10:47:32.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1807 16 374 358 0.012381 0 - 0.0803724 2024-01-11T10:47:33.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1808 16 374 358 0.0123742 0 - 0.0803724 2024-01-11T10:47:34.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1809 16 374 358 0.0123673 0 - 0.0803724 2024-01-11T10:47:35.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1810 16 374 358 0.0123605 0 - 0.0803724 2024-01-11T10:47:36.324 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1811 16 374 358 0.0123537 0 - 0.0803724 2024-01-11T10:47:37.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1812 16 374 358 0.0123469 0 - 0.0803724 2024-01-11T10:47:38.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1813 16 374 358 0.01234 0 - 0.0803724 2024-01-11T10:47:39.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1814 16 374 358 0.0123332 0 - 0.0803724 2024-01-11T10:47:40.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1815 16 374 358 0.0123265 0 - 0.0803724 2024-01-11T10:47:41.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1816 16 374 358 0.0123197 0 - 0.0803724 2024-01-11T10:47:42.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1817 16 374 358 0.0123129 0 - 0.0803724 2024-01-11T10:47:43.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1818 16 374 358 0.0123061 0 - 0.0803724 2024-01-11T10:47:44.325 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1819 16 374 358 0.0122993 0 - 0.0803724 2024-01-11T10:47:45.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:47:45.331865+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:47:45.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:47:45.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1820 16 374 358 0.0122926 0 - 0.0803724 2024-01-11T10:47:46.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1821 16 374 358 0.0122858 0 - 0.0803724 2024-01-11T10:47:47.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1822 16 374 358 0.0122791 0 - 0.0803724 2024-01-11T10:47:48.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1823 16 374 358 0.0122724 0 - 0.0803724 2024-01-11T10:47:49.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1824 16 374 358 0.0122656 0 - 0.0803724 2024-01-11T10:47:50.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1825 16 374 358 0.0122589 0 - 0.0803724 2024-01-11T10:47:51.326 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1826 16 374 358 0.0122522 0 - 0.0803724 2024-01-11T10:47:52.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1827 16 374 358 0.0122455 0 - 0.0803724 2024-01-11T10:47:53.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1828 16 374 358 0.0122388 0 - 0.0803724 2024-01-11T10:47:54.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1829 16 374 358 0.0122321 0 - 0.0803724 2024-01-11T10:47:55.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1830 16 374 358 0.0122254 0 - 0.0803724 2024-01-11T10:47:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:47:55.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:47:56.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1831 16 374 358 0.0122187 0 - 0.0803724 2024-01-11T10:47:57.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1832 16 374 358 0.0122121 0 - 0.0803724 2024-01-11T10:47:57.689 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:47:57.810 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:47:57.814+0000 7f854759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:47:58.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1833 16 374 358 0.0122054 0 - 0.0803724 2024-01-11T10:47:59.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1834 16 374 358 0.0121988 0 - 0.0803724 2024-01-11T10:48:00.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1835 16 374 358 0.0121921 0 - 0.0803724 2024-01-11T10:48:01.327 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1836 16 374 358 0.0121855 0 - 0.0803724 2024-01-11T10:48:02.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1837 16 374 358 0.0121788 0 - 0.0803724 2024-01-11T10:48:03.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1838 16 374 358 0.0121722 0 - 0.0803724 2024-01-11T10:48:04.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1839 16 374 358 0.0121656 0 - 0.0803724 2024-01-11T10:48:05.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:48:05.333985+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:48:05.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:48:05.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1840 16 374 358 0.012159 0 - 0.0803724 2024-01-11T10:48:06.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1841 16 374 358 0.0121524 0 - 0.0803724 2024-01-11T10:48:07.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1842 16 374 358 0.0121458 0 - 0.0803724 2024-01-11T10:48:08.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1843 16 374 358 0.0121392 0 - 0.0803724 2024-01-11T10:48:09.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1844 16 374 358 0.0121326 0 - 0.0803724 2024-01-11T10:48:10.328 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1845 16 374 358 0.012126 0 - 0.0803724 2024-01-11T10:48:11.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1846 16 374 358 0.0121195 0 - 0.0803724 2024-01-11T10:48:12.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1847 16 374 358 0.0121129 0 - 0.0803724 2024-01-11T10:48:13.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1848 16 374 358 0.0121063 0 - 0.0803724 2024-01-11T10:48:14.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1849 16 374 358 0.0120998 0 - 0.0803724 2024-01-11T10:48:15.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1850 16 374 358 0.0120932 0 - 0.0803724 2024-01-11T10:48:16.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1851 16 374 358 0.0120867 0 - 0.0803724 2024-01-11T10:48:17.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1852 16 374 358 0.0120802 0 - 0.0803724 2024-01-11T10:48:18.329 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1853 16 374 358 0.0120737 0 - 0.0803724 2024-01-11T10:48:19.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1854 16 374 358 0.0120672 0 - 0.0803724 2024-01-11T10:48:20.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1855 16 374 358 0.0120607 0 - 0.0803724 2024-01-11T10:48:21.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1856 16 374 358 0.0120542 0 - 0.0803724 2024-01-11T10:48:22.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1857 16 374 358 0.0120477 0 - 0.0803724 2024-01-11T10:48:23.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1858 16 374 358 0.0120412 0 - 0.0803724 2024-01-11T10:48:24.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1859 16 374 358 0.0120347 0 - 0.0803724 2024-01-11T10:48:25.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:48:25.336152+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:48:25.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:48:25.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1860 16 374 358 0.0120282 0 - 0.0803724 2024-01-11T10:48:26.330 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1861 16 374 358 0.0120218 0 - 0.0803724 2024-01-11T10:48:27.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1862 16 374 358 0.0120153 0 - 0.0803724 2024-01-11T10:48:28.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1863 16 374 358 0.0120089 0 - 0.0803724 2024-01-11T10:48:29.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1864 16 374 358 0.0120024 0 - 0.0803724 2024-01-11T10:48:30.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1865 16 374 358 0.011996 0 - 0.0803724 2024-01-11T10:48:31.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1866 16 374 358 0.0119896 0 - 0.0803724 2024-01-11T10:48:32.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1867 16 374 358 0.0119831 0 - 0.0803724 2024-01-11T10:48:33.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1868 16 374 358 0.0119767 0 - 0.0803724 2024-01-11T10:48:34.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1869 16 374 358 0.0119703 0 - 0.0803724 2024-01-11T10:48:35.331 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1870 16 374 358 0.0119639 0 - 0.0803724 2024-01-11T10:48:36.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1871 16 374 358 0.0119575 0 - 0.0803724 2024-01-11T10:48:37.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1872 16 374 358 0.0119511 0 - 0.0803724 2024-01-11T10:48:38.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1873 16 374 358 0.0119447 0 - 0.0803724 2024-01-11T10:48:39.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1874 16 374 358 0.0119384 0 - 0.0803724 2024-01-11T10:48:40.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1875 16 374 358 0.011932 0 - 0.0803724 2024-01-11T10:48:41.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1876 16 374 358 0.0119256 0 - 0.0803724 2024-01-11T10:48:42.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1877 16 374 358 0.0119193 0 - 0.0803724 2024-01-11T10:48:43.332 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1878 16 374 358 0.0119129 0 - 0.0803724 2024-01-11T10:48:44.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1879 16 374 358 0.0119066 0 - 0.0803724 2024-01-11T10:48:45.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:48:45.338241+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:48:45.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:48:45.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1880 16 374 358 0.0119003 0 - 0.0803724 2024-01-11T10:48:46.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1881 16 374 358 0.0118939 0 - 0.0803724 2024-01-11T10:48:47.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1882 16 374 358 0.0118876 0 - 0.0803724 2024-01-11T10:48:48.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1883 16 374 358 0.0118813 0 - 0.0803724 2024-01-11T10:48:49.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1884 16 374 358 0.011875 0 - 0.0803724 2024-01-11T10:48:50.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1885 16 374 358 0.0118687 0 - 0.0803724 2024-01-11T10:48:51.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1886 16 374 358 0.0118624 0 - 0.0803724 2024-01-11T10:48:52.333 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1887 16 374 358 0.0118561 0 - 0.0803724 2024-01-11T10:48:53.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1888 16 374 358 0.0118498 0 - 0.0803724 2024-01-11T10:48:54.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1889 16 374 358 0.0118436 0 - 0.0803724 2024-01-11T10:48:55.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1890 16 374 358 0.0118373 0 - 0.0803724 2024-01-11T10:48:56.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1891 16 374 358 0.011831 0 - 0.0803724 2024-01-11T10:48:57.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1892 16 374 358 0.0118248 0 - 0.0803724 2024-01-11T10:48:58.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1893 16 374 358 0.0118185 0 - 0.0803724 2024-01-11T10:48:59.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1894 16 374 358 0.0118123 0 - 0.0803724 2024-01-11T10:49:00.334 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1895 16 374 358 0.0118061 0 - 0.0803724 2024-01-11T10:49:01.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1896 16 374 358 0.0117998 0 - 0.0803724 2024-01-11T10:49:02.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1897 16 374 358 0.0117936 0 - 0.0803724 2024-01-11T10:49:03.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1898 16 374 358 0.0117874 0 - 0.0803724 2024-01-11T10:49:04.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1899 16 374 358 0.0117812 0 - 0.0803724 2024-01-11T10:49:05.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:49:05.340384+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:49:05.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:49:05.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1900 16 374 358 0.011775 0 - 0.0803724 2024-01-11T10:49:06.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1901 16 374 358 0.0117688 0 - 0.0803724 2024-01-11T10:49:07.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1902 16 374 358 0.0117626 0 - 0.0803724 2024-01-11T10:49:08.335 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1903 16 374 358 0.0117564 0 - 0.0803724 2024-01-11T10:49:09.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1904 16 374 358 0.0117503 0 - 0.0803724 2024-01-11T10:49:10.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1905 16 374 358 0.0117441 0 - 0.0803724 2024-01-11T10:49:11.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1906 16 374 358 0.0117379 0 - 0.0803724 2024-01-11T10:49:12.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1907 16 374 358 0.0117318 0 - 0.0803724 2024-01-11T10:49:13.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1908 16 374 358 0.0117256 0 - 0.0803724 2024-01-11T10:49:14.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1909 16 374 358 0.0117195 0 - 0.0803724 2024-01-11T10:49:15.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1910 16 374 358 0.0117134 0 - 0.0803724 2024-01-11T10:49:16.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1911 16 374 358 0.0117072 0 - 0.0803724 2024-01-11T10:49:17.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1912 16 374 358 0.0117011 0 - 0.0803724 2024-01-11T10:49:18.336 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1913 16 374 358 0.011695 0 - 0.0803724 2024-01-11T10:49:19.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1914 16 374 358 0.0116889 0 - 0.0803724 2024-01-11T10:49:20.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1915 16 374 358 0.0116828 0 - 0.0803724 2024-01-11T10:49:21.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1916 16 374 358 0.0116767 0 - 0.0803724 2024-01-11T10:49:22.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1917 16 374 358 0.0116706 0 - 0.0803724 2024-01-11T10:49:23.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1918 16 374 358 0.0116645 0 - 0.0803724 2024-01-11T10:49:24.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1919 16 374 358 0.0116584 0 - 0.0803724 2024-01-11T10:49:25.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:49:25.342474+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:49:25.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:49:25.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1920 16 374 358 0.0116524 0 - 0.0803724 2024-01-11T10:49:26.337 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1921 16 374 358 0.0116463 0 - 0.0803724 2024-01-11T10:49:27.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1922 16 374 358 0.0116402 0 - 0.0803724 2024-01-11T10:49:28.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1923 16 374 358 0.0116342 0 - 0.0803724 2024-01-11T10:49:29.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1924 16 374 358 0.0116281 0 - 0.0803724 2024-01-11T10:49:30.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1925 16 374 358 0.0116221 0 - 0.0803724 2024-01-11T10:49:31.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1926 16 374 358 0.0116161 0 - 0.0803724 2024-01-11T10:49:32.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1927 16 374 358 0.01161 0 - 0.0803724 2024-01-11T10:49:33.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1928 16 374 358 0.011604 0 - 0.0803724 2024-01-11T10:49:34.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1929 16 374 358 0.011598 0 - 0.0803724 2024-01-11T10:49:35.338 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1930 16 374 358 0.011592 0 - 0.0803724 2024-01-11T10:49:36.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1931 16 374 358 0.011586 0 - 0.0803724 2024-01-11T10:49:37.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1932 16 374 358 0.01158 0 - 0.0803724 2024-01-11T10:49:38.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1933 16 374 358 0.011574 0 - 0.0803724 2024-01-11T10:49:39.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1934 16 374 358 0.011568 0 - 0.0803724 2024-01-11T10:49:40.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1935 16 374 358 0.011562 0 - 0.0803724 2024-01-11T10:49:41.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1936 16 374 358 0.0115561 0 - 0.0803724 2024-01-11T10:49:42.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1937 16 374 358 0.0115501 0 - 0.0803724 2024-01-11T10:49:43.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1938 16 374 358 0.0115441 0 - 0.0803724 2024-01-11T10:49:44.339 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1939 16 374 358 0.0115382 0 - 0.0803724 2024-01-11T10:49:45.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:49:45.344549+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:49:45.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:49:45.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1940 16 374 358 0.0115322 0 - 0.0803724 2024-01-11T10:49:46.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1941 16 374 358 0.0115263 0 - 0.0803724 2024-01-11T10:49:47.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1942 16 374 358 0.0115203 0 - 0.0803724 2024-01-11T10:49:48.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1943 16 374 358 0.0115144 0 - 0.0803724 2024-01-11T10:49:49.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1944 16 374 358 0.0115085 0 - 0.0803724 2024-01-11T10:49:50.340 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1945 16 374 358 0.0115026 0 - 0.0803724 2024-01-11T10:49:51.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1946 16 374 358 0.0114967 0 - 0.0803724 2024-01-11T10:49:52.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1947 16 374 358 0.0114908 0 - 0.0803724 2024-01-11T10:49:53.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1948 16 374 358 0.0114849 0 - 0.0803724 2024-01-11T10:49:54.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1949 16 374 358 0.011479 0 - 0.0803724 2024-01-11T10:49:55.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1950 16 374 358 0.0114731 0 - 0.0803724 2024-01-11T10:49:56.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1951 16 374 358 0.0114672 0 - 0.0803724 2024-01-11T10:49:57.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1952 16 374 358 0.0114613 0 - 0.0803724 2024-01-11T10:49:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:49:57.729 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:49:58.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1953 16 374 358 0.0114555 0 - 0.0803724 2024-01-11T10:49:59.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1954 16 374 358 0.0114496 0 - 0.0803724 2024-01-11T10:49:59.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:49:59.851 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:49:59.853+0000 7f1dadd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:50:00.341 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1955 16 374 358 0.0114437 0 - 0.0803724 2024-01-11T10:50:01.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1956 16 374 358 0.0114379 0 - 0.0803724 2024-01-11T10:50:02.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1957 16 374 358 0.011432 0 - 0.0803724 2024-01-11T10:50:03.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1958 16 374 358 0.0114262 0 - 0.0803724 2024-01-11T10:50:04.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1959 16 374 358 0.0114204 0 - 0.0803724 2024-01-11T10:50:05.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:50:05.346770+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:50:05.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:50:05.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1960 16 374 358 0.0114145 0 - 0.0803724 2024-01-11T10:50:06.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1961 16 374 358 0.0114087 0 - 0.0803724 2024-01-11T10:50:07.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1962 16 374 358 0.0114029 0 - 0.0803724 2024-01-11T10:50:08.342 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1963 16 374 358 0.0113971 0 - 0.0803724 2024-01-11T10:50:09.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1964 16 374 358 0.0113913 0 - 0.0803724 2024-01-11T10:50:10.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1965 16 374 358 0.0113855 0 - 0.0803724 2024-01-11T10:50:11.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1966 16 374 358 0.0113797 0 - 0.0803724 2024-01-11T10:50:12.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1967 16 374 358 0.0113739 0 - 0.0803724 2024-01-11T10:50:13.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1968 16 374 358 0.0113681 0 - 0.0803724 2024-01-11T10:50:14.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1969 16 374 358 0.0113624 0 - 0.0803724 2024-01-11T10:50:15.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1970 16 374 358 0.0113566 0 - 0.0803724 2024-01-11T10:50:16.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1971 16 374 358 0.0113508 0 - 0.0803724 2024-01-11T10:50:17.343 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1972 16 374 358 0.0113451 0 - 0.0803724 2024-01-11T10:50:18.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1973 16 374 358 0.0113393 0 - 0.0803724 2024-01-11T10:50:19.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1974 16 374 358 0.0113336 0 - 0.0803724 2024-01-11T10:50:20.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1975 16 374 358 0.0113279 0 - 0.0803724 2024-01-11T10:50:21.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1976 16 374 358 0.0113221 0 - 0.0803724 2024-01-11T10:50:22.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1977 16 374 358 0.0113164 0 - 0.0803724 2024-01-11T10:50:23.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1978 16 374 358 0.0113107 0 - 0.0803724 2024-01-11T10:50:24.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1979 16 374 358 0.011305 0 - 0.0803724 2024-01-11T10:50:25.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:50:25.348813+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:50:25.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:50:25.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1980 16 374 358 0.0112993 0 - 0.0803724 2024-01-11T10:50:26.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1981 16 374 358 0.0112935 0 - 0.0803724 2024-01-11T10:50:27.344 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1982 16 374 358 0.0112879 0 - 0.0803724 2024-01-11T10:50:28.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1983 16 374 358 0.0112822 0 - 0.0803724 2024-01-11T10:50:29.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1984 16 374 358 0.0112765 0 - 0.0803724 2024-01-11T10:50:30.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1985 16 374 358 0.0112708 0 - 0.0803724 2024-01-11T10:50:31.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1986 16 374 358 0.0112651 0 - 0.0803724 2024-01-11T10:50:32.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1987 16 374 358 0.0112594 0 - 0.0803724 2024-01-11T10:50:33.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1988 16 374 358 0.0112538 0 - 0.0803724 2024-01-11T10:50:34.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1989 16 374 358 0.0112481 0 - 0.0803724 2024-01-11T10:50:35.345 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1990 16 374 358 0.0112425 0 - 0.0803724 2024-01-11T10:50:36.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1991 16 374 358 0.0112368 0 - 0.0803724 2024-01-11T10:50:37.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1992 16 374 358 0.0112312 0 - 0.0803724 2024-01-11T10:50:38.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1993 16 374 358 0.0112256 0 - 0.0803724 2024-01-11T10:50:39.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1994 16 374 358 0.0112199 0 - 0.0803724 2024-01-11T10:50:40.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1995 16 374 358 0.0112143 0 - 0.0803724 2024-01-11T10:50:41.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1996 16 374 358 0.0112087 0 - 0.0803724 2024-01-11T10:50:42.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1997 16 374 358 0.0112031 0 - 0.0803724 2024-01-11T10:50:43.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1998 16 374 358 0.0111975 0 - 0.0803724 2024-01-11T10:50:44.346 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 1999 16 374 358 0.0111919 0 - 0.0803724 2024-01-11T10:50:45.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:50:45.350866+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:50:45.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:50:45.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2000 16 374 358 0.0111863 0 - 0.0803724 2024-01-11T10:50:46.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2001 16 374 358 0.0111807 0 - 0.0803724 2024-01-11T10:50:47.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2002 16 374 358 0.0111751 0 - 0.0803724 2024-01-11T10:50:48.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2003 16 374 358 0.0111695 0 - 0.0803724 2024-01-11T10:50:49.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2004 16 374 358 0.0111639 0 - 0.0803724 2024-01-11T10:50:50.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2005 16 374 358 0.0111584 0 - 0.0803724 2024-01-11T10:50:51.347 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2006 16 374 358 0.0111528 0 - 0.0803724 2024-01-11T10:50:52.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2007 16 374 358 0.0111472 0 - 0.0803724 2024-01-11T10:50:53.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2008 16 374 358 0.0111417 0 - 0.0803724 2024-01-11T10:50:54.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2009 16 374 358 0.0111361 0 - 0.0803724 2024-01-11T10:50:55.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2010 16 374 358 0.0111306 0 - 0.0803724 2024-01-11T10:50:56.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2011 16 374 358 0.0111251 0 - 0.0803724 2024-01-11T10:50:57.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2012 16 374 358 0.0111195 0 - 0.0803724 2024-01-11T10:50:58.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2013 16 374 358 0.011114 0 - 0.0803724 2024-01-11T10:50:59.348 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2014 16 374 358 0.0111085 0 - 0.0803724 2024-01-11T10:51:00.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2015 16 374 358 0.011103 0 - 0.0803724 2024-01-11T10:51:01.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2016 16 374 358 0.0110975 0 - 0.0803724 2024-01-11T10:51:02.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2017 16 374 358 0.011092 0 - 0.0803724 2024-01-11T10:51:03.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2018 16 374 358 0.0110865 0 - 0.0803724 2024-01-11T10:51:04.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2019 16 374 358 0.011081 0 - 0.0803724 2024-01-11T10:51:05.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:51:05.353153+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:51:05.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:51:05.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2020 16 374 358 0.0110755 0 - 0.0803724 2024-01-11T10:51:06.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2021 16 374 358 0.01107 0 - 0.0803724 2024-01-11T10:51:07.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2022 16 374 358 0.0110646 0 - 0.0803724 2024-01-11T10:51:08.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2023 16 374 358 0.0110591 0 - 0.0803724 2024-01-11T10:51:09.349 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2024 16 374 358 0.0110536 0 - 0.0803724 2024-01-11T10:51:10.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2025 16 374 358 0.0110482 0 - 0.0803724 2024-01-11T10:51:11.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2026 16 374 358 0.0110427 0 - 0.0803724 2024-01-11T10:51:12.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2027 16 374 358 0.0110373 0 - 0.0803724 2024-01-11T10:51:13.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2028 16 374 358 0.0110318 0 - 0.0803724 2024-01-11T10:51:14.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2029 16 374 358 0.0110264 0 - 0.0803724 2024-01-11T10:51:15.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2030 16 374 358 0.0110209 0 - 0.0803724 2024-01-11T10:51:16.350 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2031 16 374 358 0.0110155 0 - 0.0803724 2024-01-11T10:51:17.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2032 16 374 358 0.0110101 0 - 0.0803724 2024-01-11T10:51:18.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2033 16 374 358 0.0110047 0 - 0.0803724 2024-01-11T10:51:19.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2034 16 374 358 0.0109993 0 - 0.0803724 2024-01-11T10:51:20.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2035 16 374 358 0.0109939 0 - 0.0803724 2024-01-11T10:51:21.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2036 16 374 358 0.0109885 0 - 0.0803724 2024-01-11T10:51:22.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2037 16 374 358 0.0109831 0 - 0.0803724 2024-01-11T10:51:23.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2038 16 374 358 0.0109777 0 - 0.0803724 2024-01-11T10:51:24.351 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2039 16 374 358 0.0109723 0 - 0.0803724 2024-01-11T10:51:25.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:51:25.355381+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:51:25.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:51:25.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2040 16 374 358 0.0109669 0 - 0.0803724 2024-01-11T10:51:26.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2041 16 374 358 0.0109615 0 - 0.0803724 2024-01-11T10:51:27.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2042 16 374 358 0.0109562 0 - 0.0803724 2024-01-11T10:51:28.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2043 16 374 358 0.0109508 0 - 0.0803724 2024-01-11T10:51:29.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2044 16 374 358 0.0109455 0 - 0.0803724 2024-01-11T10:51:30.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2045 16 374 358 0.0109401 0 - 0.0803724 2024-01-11T10:51:31.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2046 16 374 358 0.0109348 0 - 0.0803724 2024-01-11T10:51:32.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2047 16 374 358 0.0109294 0 - 0.0803724 2024-01-11T10:51:33.352 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2048 16 374 358 0.0109241 0 - 0.0803724 2024-01-11T10:51:34.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2049 16 374 358 0.0109188 0 - 0.0803724 2024-01-11T10:51:35.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2050 16 374 358 0.0109134 0 - 0.0803724 2024-01-11T10:51:36.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2051 16 374 358 0.0109081 0 - 0.0803724 2024-01-11T10:51:37.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2052 16 374 358 0.0109028 0 - 0.0803724 2024-01-11T10:51:38.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2053 16 374 358 0.0108975 0 - 0.0803724 2024-01-11T10:51:39.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2054 16 374 358 0.0108922 0 - 0.0803724 2024-01-11T10:51:40.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2055 16 374 358 0.0108869 0 - 0.0803724 2024-01-11T10:51:41.353 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2056 16 374 358 0.0108816 0 - 0.0803724 2024-01-11T10:51:42.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2057 16 374 358 0.0108763 0 - 0.0803724 2024-01-11T10:51:43.354 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2058 16 374 358 0.010871 0 - 0.0803724 2024-01-11T10:51:44.354 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2059 16 374 358 0.0108657 0 - 0.0803724 2024-01-11T10:51:45.354 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:51:45.357728+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:51:45.354 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:51:45.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2060 16 374 358 0.0108604 0 - 0.0803724 2024-01-11T10:51:46.354 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2061 16 374 358 0.0108552 0 - 0.0803724 2024-01-11T10:51:47.354 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2062 16 374 358 0.0108499 0 - 0.0803724 2024-01-11T10:51:48.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2063 16 374 358 0.0108447 0 - 0.0803724 2024-01-11T10:51:49.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2064 16 374 358 0.0108394 0 - 0.0803724 2024-01-11T10:51:50.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2065 16 374 358 0.0108341 0 - 0.0803724 2024-01-11T10:51:51.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2066 16 374 358 0.0108289 0 - 0.0803724 2024-01-11T10:51:52.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2067 16 374 358 0.0108237 0 - 0.0803724 2024-01-11T10:51:53.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2068 16 374 358 0.0108184 0 - 0.0803724 2024-01-11T10:51:54.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2069 16 374 358 0.0108132 0 - 0.0803724 2024-01-11T10:51:55.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2070 16 374 358 0.010808 0 - 0.0803724 2024-01-11T10:51:56.355 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2071 16 374 358 0.0108028 0 - 0.0803724 2024-01-11T10:51:57.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2072 16 374 358 0.0107975 0 - 0.0803724 2024-01-11T10:51:58.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2073 16 374 358 0.0107923 0 - 0.0803724 2024-01-11T10:51:59.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2074 16 374 358 0.0107871 0 - 0.0803724 2024-01-11T10:51:59.770 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:51:59.770 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:52:00.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2075 16 374 358 0.0107819 0 - 0.0803724 2024-01-11T10:52:01.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2076 16 374 358 0.0107767 0 - 0.0803724 2024-01-11T10:52:01.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:52:01.890 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:52:01.890+0000 7f3929b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:52:02.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2077 16 374 358 0.0107716 0 - 0.0803724 2024-01-11T10:52:03.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2078 16 374 358 0.0107664 0 - 0.0803724 2024-01-11T10:52:04.356 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2079 16 374 358 0.0107612 0 - 0.0803724 2024-01-11T10:52:05.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:52:05.360069+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:52:05.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:52:05.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2080 16 374 358 0.010756 0 - 0.0803724 2024-01-11T10:52:06.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2081 16 374 358 0.0107508 0 - 0.0803724 2024-01-11T10:52:07.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2082 16 374 358 0.0107457 0 - 0.0803724 2024-01-11T10:52:08.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2083 16 374 358 0.0107405 0 - 0.0803724 2024-01-11T10:52:09.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2084 16 374 358 0.0107354 0 - 0.0803724 2024-01-11T10:52:10.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2085 16 374 358 0.0107302 0 - 0.0803724 2024-01-11T10:52:11.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2086 16 374 358 0.0107251 0 - 0.0803724 2024-01-11T10:52:12.357 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2087 16 374 358 0.0107199 0 - 0.0803724 2024-01-11T10:52:13.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2088 16 374 358 0.0107148 0 - 0.0803724 2024-01-11T10:52:14.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2089 16 374 358 0.0107097 0 - 0.0803724 2024-01-11T10:52:15.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2090 16 374 358 0.0107046 0 - 0.0803724 2024-01-11T10:52:16.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2091 16 374 358 0.0106994 0 - 0.0803724 2024-01-11T10:52:17.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2092 16 374 358 0.0106943 0 - 0.0803724 2024-01-11T10:52:18.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2093 16 374 358 0.0106892 0 - 0.0803724 2024-01-11T10:52:19.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2094 16 374 358 0.0106841 0 - 0.0803724 2024-01-11T10:52:20.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2095 16 374 358 0.010679 0 - 0.0803724 2024-01-11T10:52:21.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2096 16 374 358 0.0106739 0 - 0.0803724 2024-01-11T10:52:22.358 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2097 16 374 358 0.0106688 0 - 0.0803724 2024-01-11T10:52:23.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2098 16 374 358 0.0106637 0 - 0.0803724 2024-01-11T10:52:24.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2099 16 374 358 0.0106587 0 - 0.0803724 2024-01-11T10:52:25.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:52:25.362153+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:52:25.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:52:25.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2100 16 374 358 0.0106536 0 - 0.0803724 2024-01-11T10:52:26.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2101 16 374 358 0.0106485 0 - 0.0803724 2024-01-11T10:52:27.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2102 16 374 358 0.0106434 0 - 0.0803724 2024-01-11T10:52:28.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2103 16 374 358 0.0106384 0 - 0.0803724 2024-01-11T10:52:29.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2104 16 374 358 0.0106333 0 - 0.0803724 2024-01-11T10:52:30.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2105 16 374 358 0.0106283 0 - 0.0803724 2024-01-11T10:52:31.359 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2106 16 374 358 0.0106232 0 - 0.0803724 2024-01-11T10:52:32.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2107 16 374 358 0.0106182 0 - 0.0803724 2024-01-11T10:52:33.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2108 16 374 358 0.0106131 0 - 0.0803724 2024-01-11T10:52:34.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2109 16 374 358 0.0106081 0 - 0.0803724 2024-01-11T10:52:35.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2110 16 374 358 0.0106031 0 - 0.0803724 2024-01-11T10:52:36.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2111 16 374 358 0.0105981 0 - 0.0803724 2024-01-11T10:52:37.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2112 16 374 358 0.010593 0 - 0.0803724 2024-01-11T10:52:38.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2113 16 374 358 0.010588 0 - 0.0803724 2024-01-11T10:52:39.360 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2114 16 374 358 0.010583 0 - 0.0803724 2024-01-11T10:52:40.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2115 16 374 358 0.010578 0 - 0.0803724 2024-01-11T10:52:41.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2116 16 374 358 0.010573 0 - 0.0803724 2024-01-11T10:52:42.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2117 16 374 358 0.010568 0 - 0.0803724 2024-01-11T10:52:43.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2118 16 374 358 0.010563 0 - 0.0803724 2024-01-11T10:52:44.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2119 16 374 358 0.0105581 0 - 0.0803724 2024-01-11T10:52:45.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:52:45.364268+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:52:45.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:52:45.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2120 16 374 358 0.0105531 0 - 0.0803724 2024-01-11T10:52:46.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2121 16 374 358 0.0105481 0 - 0.0803724 2024-01-11T10:52:47.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2122 16 374 358 0.0105431 0 - 0.0803724 2024-01-11T10:52:48.361 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2123 16 374 358 0.0105382 0 - 0.0803724 2024-01-11T10:52:49.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2124 16 374 358 0.0105332 0 - 0.0803724 2024-01-11T10:52:50.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2125 16 374 358 0.0105282 0 - 0.0803724 2024-01-11T10:52:51.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2126 16 374 358 0.0105233 0 - 0.0803724 2024-01-11T10:52:52.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2127 16 374 358 0.0105183 0 - 0.0803724 2024-01-11T10:52:53.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2128 16 374 358 0.0105134 0 - 0.0803724 2024-01-11T10:52:54.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2129 16 374 358 0.0105085 0 - 0.0803724 2024-01-11T10:52:55.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2130 16 374 358 0.0105035 0 - 0.0803724 2024-01-11T10:52:56.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2131 16 374 358 0.0104986 0 - 0.0803724 2024-01-11T10:52:57.362 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2132 16 374 358 0.0104937 0 - 0.0803724 2024-01-11T10:52:58.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2133 16 374 358 0.0104888 0 - 0.0803724 2024-01-11T10:52:59.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2134 16 374 358 0.0104838 0 - 0.0803724 2024-01-11T10:53:00.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2135 16 374 358 0.0104789 0 - 0.0803724 2024-01-11T10:53:01.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2136 16 374 358 0.010474 0 - 0.0803724 2024-01-11T10:53:02.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2137 16 374 358 0.0104691 0 - 0.0803724 2024-01-11T10:53:03.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2138 16 374 358 0.0104642 0 - 0.0803724 2024-01-11T10:53:04.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2139 16 374 358 0.0104593 0 - 0.0803724 2024-01-11T10:53:05.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:53:05.366490+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:53:05.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:53:05.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2140 16 374 358 0.0104544 0 - 0.0803724 2024-01-11T10:53:06.363 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2141 16 374 358 0.0104496 0 - 0.0803724 2024-01-11T10:53:07.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2142 16 374 358 0.0104447 0 - 0.0803724 2024-01-11T10:53:08.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2143 16 374 358 0.0104398 0 - 0.0803724 2024-01-11T10:53:09.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2144 16 374 358 0.0104349 0 - 0.0803724 2024-01-11T10:53:10.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2145 16 374 358 0.0104301 0 - 0.0803724 2024-01-11T10:53:11.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2146 16 374 358 0.0104252 0 - 0.0803724 2024-01-11T10:53:12.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2147 16 374 358 0.0104204 0 - 0.0803724 2024-01-11T10:53:13.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2148 16 374 358 0.0104155 0 - 0.0803724 2024-01-11T10:53:14.364 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2149 16 374 358 0.0104107 0 - 0.0803724 2024-01-11T10:53:15.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2150 16 374 358 0.0104058 0 - 0.0803724 2024-01-11T10:53:16.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2151 16 374 358 0.010401 0 - 0.0803724 2024-01-11T10:53:17.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2152 16 374 358 0.0103962 0 - 0.0803724 2024-01-11T10:53:18.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2153 16 374 358 0.0103913 0 - 0.0803724 2024-01-11T10:53:19.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2154 16 374 358 0.0103865 0 - 0.0803724 2024-01-11T10:53:20.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2155 16 374 358 0.0103817 0 - 0.0803724 2024-01-11T10:53:21.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2156 16 374 358 0.0103769 0 - 0.0803724 2024-01-11T10:53:22.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2157 16 374 358 0.0103721 0 - 0.0803724 2024-01-11T10:53:23.365 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2158 16 374 358 0.0103672 0 - 0.0803724 2024-01-11T10:53:24.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2159 16 374 358 0.0103624 0 - 0.0803724 2024-01-11T10:53:25.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:53:25.368527+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:53:25.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:53:25.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2160 16 374 358 0.0103576 0 - 0.0803724 2024-01-11T10:53:26.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2161 16 374 358 0.0103529 0 - 0.0803724 2024-01-11T10:53:27.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2162 16 374 358 0.0103481 0 - 0.0803724 2024-01-11T10:53:28.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2163 16 374 358 0.0103433 0 - 0.0803724 2024-01-11T10:53:29.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2164 16 374 358 0.0103385 0 - 0.0803724 2024-01-11T10:53:30.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2165 16 374 358 0.0103337 0 - 0.0803724 2024-01-11T10:53:31.366 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2166 16 374 358 0.010329 0 - 0.0803724 2024-01-11T10:53:32.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2167 16 374 358 0.0103242 0 - 0.0803724 2024-01-11T10:53:33.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2168 16 374 358 0.0103194 0 - 0.0803724 2024-01-11T10:53:34.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2169 16 374 358 0.0103147 0 - 0.0803724 2024-01-11T10:53:35.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2170 16 374 358 0.0103099 0 - 0.0803724 2024-01-11T10:53:36.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2171 16 374 358 0.0103052 0 - 0.0803724 2024-01-11T10:53:37.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2172 16 374 358 0.0103004 0 - 0.0803724 2024-01-11T10:53:38.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2173 16 374 358 0.0102957 0 - 0.0803724 2024-01-11T10:53:39.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2174 16 374 358 0.0102909 0 - 0.0803724 2024-01-11T10:53:40.367 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2175 16 374 358 0.0102862 0 - 0.0803724 2024-01-11T10:53:41.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2176 16 374 358 0.0102815 0 - 0.0803724 2024-01-11T10:53:42.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2177 16 374 358 0.0102768 0 - 0.0803724 2024-01-11T10:53:43.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2178 16 374 358 0.010272 0 - 0.0803724 2024-01-11T10:53:44.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2179 16 374 358 0.0102673 0 - 0.0803724 2024-01-11T10:53:45.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:53:45.370683+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:53:45.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:53:45.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2180 16 374 358 0.0102626 0 - 0.0803724 2024-01-11T10:53:46.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2181 16 374 358 0.0102579 0 - 0.0803724 2024-01-11T10:53:47.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2182 16 374 358 0.0102532 0 - 0.0803724 2024-01-11T10:53:48.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2183 16 374 358 0.0102485 0 - 0.0803724 2024-01-11T10:53:49.368 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2184 16 374 358 0.0102438 0 - 0.0803724 2024-01-11T10:53:50.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2185 16 374 358 0.0102391 0 - 0.0803724 2024-01-11T10:53:51.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2186 16 374 358 0.0102345 0 - 0.0803724 2024-01-11T10:53:52.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2187 16 374 358 0.0102298 0 - 0.0803724 2024-01-11T10:53:53.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2188 16 374 358 0.0102251 0 - 0.0803724 2024-01-11T10:53:54.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2189 16 374 358 0.0102204 0 - 0.0803724 2024-01-11T10:53:55.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2190 16 374 358 0.0102158 0 - 0.0803724 2024-01-11T10:53:56.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2191 16 374 358 0.0102111 0 - 0.0803724 2024-01-11T10:53:57.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2192 16 374 358 0.0102064 0 - 0.0803724 2024-01-11T10:53:58.369 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2193 16 374 358 0.0102018 0 - 0.0803724 2024-01-11T10:53:59.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2194 16 374 358 0.0101971 0 - 0.0803724 2024-01-11T10:54:00.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2195 16 374 358 0.0101925 0 - 0.0803724 2024-01-11T10:54:01.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2196 16 374 358 0.0101879 0 - 0.0803724 2024-01-11T10:54:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:54:01.811 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:54:02.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2197 16 374 358 0.0101832 0 - 0.0803724 2024-01-11T10:54:03.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2198 16 374 358 0.0101786 0 - 0.0803724 2024-01-11T10:54:03.812 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:54:03.930 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:54:03.931+0000 7ff9e80de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:54:04.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2199 16 374 358 0.010174 0 - 0.0803724 2024-01-11T10:54:05.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:54:05.372870+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:54:05.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:54:05.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2200 16 374 358 0.0101693 0 - 0.0803724 2024-01-11T10:54:06.370 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2201 16 374 358 0.0101647 0 - 0.0803724 2024-01-11T10:54:07.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2202 16 374 358 0.0101601 0 - 0.0803724 2024-01-11T10:54:08.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2203 16 374 358 0.0101555 0 - 0.0803724 2024-01-11T10:54:09.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2204 16 374 358 0.0101509 0 - 0.0803724 2024-01-11T10:54:10.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2205 16 374 358 0.0101463 0 - 0.0803724 2024-01-11T10:54:11.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2206 16 374 358 0.0101417 0 - 0.0803724 2024-01-11T10:54:12.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2207 16 374 358 0.0101371 0 - 0.0803724 2024-01-11T10:54:13.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2208 16 374 358 0.0101325 0 - 0.0803724 2024-01-11T10:54:14.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2209 16 374 358 0.0101279 0 - 0.0803724 2024-01-11T10:54:15.371 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2210 16 374 358 0.0101233 0 - 0.0803724 2024-01-11T10:54:16.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2211 16 374 358 0.0101187 0 - 0.0803724 2024-01-11T10:54:17.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2212 16 374 358 0.0101142 0 - 0.0803724 2024-01-11T10:54:18.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2213 16 374 358 0.0101096 0 - 0.0803724 2024-01-11T10:54:19.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2214 16 374 358 0.010105 0 - 0.0803724 2024-01-11T10:54:20.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2215 16 374 358 0.0101005 0 - 0.0803724 2024-01-11T10:54:21.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2216 16 374 358 0.0100959 0 - 0.0803724 2024-01-11T10:54:22.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2217 16 374 358 0.0100914 0 - 0.0803724 2024-01-11T10:54:23.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2218 16 374 358 0.0100868 0 - 0.0803724 2024-01-11T10:54:24.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2219 16 374 358 0.0100823 0 - 0.0803724 2024-01-11T10:54:25.372 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:54:25.375004+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:54:25.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:54:25.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2220 16 374 358 0.0100777 0 - 0.0803724 2024-01-11T10:54:26.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2221 16 374 358 0.0100732 0 - 0.0803724 2024-01-11T10:54:27.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2222 16 374 358 0.0100686 0 - 0.0803724 2024-01-11T10:54:28.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2223 16 374 358 0.0100641 0 - 0.0803724 2024-01-11T10:54:29.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2224 16 374 358 0.0100596 0 - 0.0803724 2024-01-11T10:54:30.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2225 16 374 358 0.0100551 0 - 0.0803724 2024-01-11T10:54:31.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2226 16 374 358 0.0100506 0 - 0.0803724 2024-01-11T10:54:32.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2227 16 374 358 0.010046 0 - 0.0803724 2024-01-11T10:54:33.373 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2228 16 374 358 0.0100415 0 - 0.0803724 2024-01-11T10:54:34.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2229 16 374 358 0.010037 0 - 0.0803724 2024-01-11T10:54:35.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2230 16 374 358 0.0100325 0 - 0.0803724 2024-01-11T10:54:36.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2231 16 374 358 0.010028 0 - 0.0803724 2024-01-11T10:54:37.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2232 16 374 358 0.0100235 0 - 0.0803724 2024-01-11T10:54:38.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2233 16 374 358 0.010019 0 - 0.0803724 2024-01-11T10:54:39.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2234 16 374 358 0.0100146 0 - 0.0803724 2024-01-11T10:54:40.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2235 16 374 358 0.0100101 0 - 0.0803724 2024-01-11T10:54:41.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2236 16 374 358 0.0100056 0 - 0.0803724 2024-01-11T10:54:42.374 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2237 16 374 358 0.0100011 0 - 0.0803724 2024-01-11T10:54:43.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2238 16 374 358 0.00999666 0 - 0.0803724 2024-01-11T10:54:44.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2239 16 374 358 0.0099922 0 - 0.0803724 2024-01-11T10:54:45.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:54:45.377105+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:54:45.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:54:45.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2240 16 374 358 0.00998773 0 - 0.0803724 2024-01-11T10:54:46.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2241 16 374 358 0.00998328 0 - 0.0803724 2024-01-11T10:54:47.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2242 16 374 358 0.00997882 0 - 0.0803724 2024-01-11T10:54:48.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2243 16 374 358 0.00997438 0 - 0.0803724 2024-01-11T10:54:49.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2244 16 374 358 0.00996993 0 - 0.0803724 2024-01-11T10:54:50.375 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2245 16 374 358 0.00996549 0 - 0.0803724 2024-01-11T10:54:51.376 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2246 16 374 358 0.00996105 0 - 0.0803724 2024-01-11T10:54:52.376 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2247 16 374 358 0.00995662 0 - 0.0803724 2024-01-11T10:54:53.376 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2248 16 374 358 0.00995219 0 - 0.0803724 2024-01-11T10:54:54.376 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2249 16 374 358 0.00994777 0 - 0.0803724 2024-01-11T10:54:55.376 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2250 16 374 358 0.00994334 0 - 0.0803724 2024-01-11T10:54:56.376 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2251 16 374 358 0.00993893 0 - 0.0803724 2024-01-11T10:54:57.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2252 16 374 358 0.00993451 0 - 0.0803724 2024-01-11T10:54:58.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2253 16 374 358 0.0099301 0 - 0.0803724 2024-01-11T10:54:59.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2254 16 374 358 0.0099257 0 - 0.0803724 2024-01-11T10:55:00.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2255 16 374 358 0.0099213 0 - 0.0803724 2024-01-11T10:55:01.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2256 16 374 358 0.0099169 0 - 0.0803724 2024-01-11T10:55:02.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2257 16 374 358 0.00991251 0 - 0.0803724 2024-01-11T10:55:03.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2258 16 374 358 0.00990812 0 - 0.0803724 2024-01-11T10:55:04.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2259 16 374 358 0.00990373 0 - 0.0803724 2024-01-11T10:55:05.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:55:05.379451+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:55:05.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:55:05.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2260 16 374 358 0.00989935 0 - 0.0803724 2024-01-11T10:55:06.377 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2261 16 374 358 0.00989497 0 - 0.0803724 2024-01-11T10:55:07.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2262 16 374 358 0.00989059 0 - 0.0803724 2024-01-11T10:55:08.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2263 16 374 358 0.00988622 0 - 0.0803724 2024-01-11T10:55:09.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2264 16 374 358 0.00988186 0 - 0.0803724 2024-01-11T10:55:10.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2265 16 374 358 0.00987749 0 - 0.0803724 2024-01-11T10:55:11.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2266 16 374 358 0.00987313 0 - 0.0803724 2024-01-11T10:55:12.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2267 16 374 358 0.00986878 0 - 0.0803724 2024-01-11T10:55:13.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2268 16 374 358 0.00986443 0 - 0.0803724 2024-01-11T10:55:14.378 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2269 16 374 358 0.00986008 0 - 0.0803724 2024-01-11T10:55:15.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2270 16 374 358 0.00985574 0 - 0.0803724 2024-01-11T10:55:16.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2271 16 374 358 0.0098514 0 - 0.0803724 2024-01-11T10:55:17.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2272 16 374 358 0.00984706 0 - 0.0803724 2024-01-11T10:55:18.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2273 16 374 358 0.00984273 0 - 0.0803724 2024-01-11T10:55:19.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2274 16 374 358 0.0098384 0 - 0.0803724 2024-01-11T10:55:20.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2275 16 374 358 0.00983408 0 - 0.0803724 2024-01-11T10:55:21.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2276 16 374 358 0.00982976 0 - 0.0803724 2024-01-11T10:55:22.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2277 16 374 358 0.00982544 0 - 0.0803724 2024-01-11T10:55:23.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2278 16 374 358 0.00982113 0 - 0.0803724 2024-01-11T10:55:24.379 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2279 16 374 358 0.00981682 0 - 0.0803724 2024-01-11T10:55:25.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:55:25.381675+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:55:25.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:55:25.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2280 16 374 358 0.00981251 0 - 0.0803724 2024-01-11T10:55:26.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2281 16 374 358 0.00980821 0 - 0.0803724 2024-01-11T10:55:27.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2282 16 374 358 0.00980391 0 - 0.0803724 2024-01-11T10:55:28.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2283 16 374 358 0.00979962 0 - 0.0803724 2024-01-11T10:55:29.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2284 16 374 358 0.00979533 0 - 0.0803724 2024-01-11T10:55:30.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2285 16 374 358 0.00979104 0 - 0.0803724 2024-01-11T10:55:31.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2286 16 374 358 0.00978676 0 - 0.0803724 2024-01-11T10:55:32.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2287 16 374 358 0.00978248 0 - 0.0803724 2024-01-11T10:55:33.380 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2288 16 374 358 0.0097782 0 - 0.0803724 2024-01-11T10:55:34.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2289 16 374 358 0.00977393 0 - 0.0803724 2024-01-11T10:55:35.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2290 16 374 358 0.00976966 0 - 0.0803724 2024-01-11T10:55:36.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2291 16 374 358 0.0097654 0 - 0.0803724 2024-01-11T10:55:37.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2292 16 374 358 0.00976114 0 - 0.0803724 2024-01-11T10:55:38.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2293 16 374 358 0.00975688 0 - 0.0803724 2024-01-11T10:55:39.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2294 16 374 358 0.00975263 0 - 0.0803724 2024-01-11T10:55:40.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2295 16 374 358 0.00974838 0 - 0.0803724 2024-01-11T10:55:41.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2296 16 374 358 0.00974413 0 - 0.0803724 2024-01-11T10:55:42.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2297 16 374 358 0.00973989 0 - 0.0803724 2024-01-11T10:55:43.381 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2298 16 374 358 0.00973565 0 - 0.0803724 2024-01-11T10:55:44.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2299 16 374 358 0.00973142 0 - 0.0803724 2024-01-11T10:55:45.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:55:45.383717+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:55:45.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:55:45.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2300 16 374 358 0.00972718 0 - 0.0803724 2024-01-11T10:55:46.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2301 16 374 358 0.00972296 0 - 0.0803724 2024-01-11T10:55:47.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2302 16 374 358 0.00971873 0 - 0.0803724 2024-01-11T10:55:48.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2303 16 374 358 0.00971451 0 - 0.0803724 2024-01-11T10:55:49.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2304 16 374 358 0.0097103 0 - 0.0803724 2024-01-11T10:55:50.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2305 16 374 358 0.00970608 0 - 0.0803724 2024-01-11T10:55:51.382 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2306 16 374 358 0.00970188 0 - 0.0803724 2024-01-11T10:55:52.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2307 16 374 358 0.00969767 0 - 0.0803724 2024-01-11T10:55:53.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2308 16 374 358 0.00969347 0 - 0.0803724 2024-01-11T10:55:54.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2309 16 374 358 0.00968927 0 - 0.0803724 2024-01-11T10:55:55.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2310 16 374 358 0.00968508 0 - 0.0803724 2024-01-11T10:55:56.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2311 16 374 358 0.00968089 0 - 0.0803724 2024-01-11T10:55:57.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2312 16 374 358 0.0096767 0 - 0.0803724 2024-01-11T10:55:58.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2313 16 374 358 0.00967251 0 - 0.0803724 2024-01-11T10:55:59.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2314 16 374 358 0.00966833 0 - 0.0803724 2024-01-11T10:56:00.383 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2315 16 374 358 0.00966416 0 - 0.0803724 2024-01-11T10:56:01.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2316 16 374 358 0.00965999 0 - 0.0803724 2024-01-11T10:56:02.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2317 16 374 358 0.00965582 0 - 0.0803724 2024-01-11T10:56:03.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2318 16 374 358 0.00965165 0 - 0.0803724 2024-01-11T10:56:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:56:03.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:56:04.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2319 16 374 358 0.00964749 0 - 0.0803724 2024-01-11T10:56:05.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:56:05.385810+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:56:05.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:56:05.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2320 16 374 358 0.00964333 0 - 0.0803724 2024-01-11T10:56:05.852 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T10:56:05.970 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:56:05.970+0000 7f64fd71e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:56:06.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2321 16 374 358 0.00963918 0 - 0.0803724 2024-01-11T10:56:07.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2322 16 374 358 0.00963502 0 - 0.0803724 2024-01-11T10:56:08.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2323 16 374 358 0.00963088 0 - 0.0803724 2024-01-11T10:56:09.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2324 16 374 358 0.00962673 0 - 0.0803724 2024-01-11T10:56:10.384 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2325 16 374 358 0.00962259 0 - 0.0803724 2024-01-11T10:56:11.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2326 16 374 358 0.00961846 0 - 0.0803724 2024-01-11T10:56:12.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2327 16 374 358 0.00961432 0 - 0.0803724 2024-01-11T10:56:13.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2328 16 374 358 0.00961019 0 - 0.0803724 2024-01-11T10:56:14.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2329 16 374 358 0.00960607 0 - 0.0803724 2024-01-11T10:56:15.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2330 16 374 358 0.00960194 0 - 0.0803724 2024-01-11T10:56:16.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2331 16 374 358 0.00959782 0 - 0.0803724 2024-01-11T10:56:17.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2332 16 374 358 0.00959371 0 - 0.0803724 2024-01-11T10:56:18.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2333 16 374 358 0.0095896 0 - 0.0803724 2024-01-11T10:56:19.385 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2334 16 374 358 0.00958549 0 - 0.0803724 2024-01-11T10:56:20.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2335 16 374 358 0.00958138 0 - 0.0803724 2024-01-11T10:56:21.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2336 16 374 358 0.00957728 0 - 0.0803724 2024-01-11T10:56:22.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2337 16 374 358 0.00957318 0 - 0.0803724 2024-01-11T10:56:23.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2338 16 374 358 0.00956909 0 - 0.0803724 2024-01-11T10:56:24.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2339 16 374 358 0.009565 0 - 0.0803724 2024-01-11T10:56:25.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:56:25.387887+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:56:25.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:56:25.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2340 16 374 358 0.00956091 0 - 0.0803724 2024-01-11T10:56:26.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2341 16 374 358 0.00955683 0 - 0.0803724 2024-01-11T10:56:27.386 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2342 16 374 358 0.00955274 0 - 0.0803724 2024-01-11T10:56:28.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2343 16 374 358 0.00954867 0 - 0.0803724 2024-01-11T10:56:29.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2344 16 374 358 0.00954459 0 - 0.0803724 2024-01-11T10:56:30.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2345 16 374 358 0.00954052 0 - 0.0803724 2024-01-11T10:56:31.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2346 16 374 358 0.00953646 0 - 0.0803724 2024-01-11T10:56:32.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2347 16 374 358 0.00953239 0 - 0.0803724 2024-01-11T10:56:33.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2348 16 374 358 0.00952833 0 - 0.0803724 2024-01-11T10:56:34.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2349 16 374 358 0.00952428 0 - 0.0803724 2024-01-11T10:56:35.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2350 16 374 358 0.00952022 0 - 0.0803724 2024-01-11T10:56:36.387 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2351 16 374 358 0.00951618 0 - 0.0803724 2024-01-11T10:56:37.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2352 16 374 358 0.00951213 0 - 0.0803724 2024-01-11T10:56:38.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2353 16 374 358 0.00950809 0 - 0.0803724 2024-01-11T10:56:39.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2354 16 374 358 0.00950405 0 - 0.0803724 2024-01-11T10:56:40.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2355 16 374 358 0.00950001 0 - 0.0803724 2024-01-11T10:56:41.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2356 16 374 358 0.00949598 0 - 0.0803724 2024-01-11T10:56:42.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2357 16 374 358 0.00949195 0 - 0.0803724 2024-01-11T10:56:43.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2358 16 374 358 0.00948793 0 - 0.0803724 2024-01-11T10:56:44.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2359 16 374 358 0.0094839 0 - 0.0803724 2024-01-11T10:56:45.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:56:45.389898+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:56:45.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:56:45.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2360 16 374 358 0.00947989 0 - 0.0803724 2024-01-11T10:56:46.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2361 16 374 358 0.00947587 0 - 0.0803724 2024-01-11T10:56:47.388 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2362 16 374 358 0.00947186 0 - 0.0803724 2024-01-11T10:56:48.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2363 16 374 358 0.00946785 0 - 0.0803724 2024-01-11T10:56:49.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2364 16 374 358 0.00946385 0 - 0.0803724 2024-01-11T10:56:50.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2365 16 374 358 0.00945984 0 - 0.0803724 2024-01-11T10:56:51.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2366 16 374 358 0.00945585 0 - 0.0803724 2024-01-11T10:56:52.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2367 16 374 358 0.00945185 0 - 0.0803724 2024-01-11T10:56:53.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2368 16 374 358 0.00944786 0 - 0.0803724 2024-01-11T10:56:54.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2369 16 374 358 0.00944387 0 - 0.0803724 2024-01-11T10:56:55.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2370 16 374 358 0.00943989 0 - 0.0803724 2024-01-11T10:56:56.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2371 16 374 358 0.0094359 0 - 0.0803724 2024-01-11T10:56:57.389 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2372 16 374 358 0.00943193 0 - 0.0803724 2024-01-11T10:56:58.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2373 16 374 358 0.00942795 0 - 0.0803724 2024-01-11T10:56:59.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2374 16 374 358 0.00942398 0 - 0.0803724 2024-01-11T10:57:00.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2375 16 374 358 0.00942001 0 - 0.0803724 2024-01-11T10:57:01.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2376 16 374 358 0.00941605 0 - 0.0803724 2024-01-11T10:57:02.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2377 16 374 358 0.00941209 0 - 0.0803724 2024-01-11T10:57:03.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2378 16 374 358 0.00940813 0 - 0.0803724 2024-01-11T10:57:04.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2379 16 374 358 0.00940417 0 - 0.0803724 2024-01-11T10:57:05.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:57:05.391934+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:57:05.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:57:05.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2380 16 374 358 0.00940022 0 - 0.0803724 2024-01-11T10:57:06.390 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2381 16 374 358 0.00939628 0 - 0.0803724 2024-01-11T10:57:07.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2382 16 374 358 0.00939233 0 - 0.0803724 2024-01-11T10:57:08.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2383 16 374 358 0.00938839 0 - 0.0803724 2024-01-11T10:57:09.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2384 16 374 358 0.00938445 0 - 0.0803724 2024-01-11T10:57:10.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2385 16 374 358 0.00938052 0 - 0.0803724 2024-01-11T10:57:11.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2386 16 374 358 0.00937658 0 - 0.0803724 2024-01-11T10:57:12.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2387 16 374 358 0.00937266 0 - 0.0803724 2024-01-11T10:57:13.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2388 16 374 358 0.00936873 0 - 0.0803724 2024-01-11T10:57:14.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2389 16 374 358 0.00936481 0 - 0.0803724 2024-01-11T10:57:15.391 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2390 16 374 358 0.00936089 0 - 0.0803724 2024-01-11T10:57:16.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2391 16 374 358 0.00935698 0 - 0.0803724 2024-01-11T10:57:17.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2392 16 374 358 0.00935307 0 - 0.0803724 2024-01-11T10:57:18.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2393 16 374 358 0.00934916 0 - 0.0803724 2024-01-11T10:57:19.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2394 16 374 358 0.00934525 0 - 0.0803724 2024-01-11T10:57:20.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2395 16 374 358 0.00934135 0 - 0.0803724 2024-01-11T10:57:21.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2396 16 374 358 0.00933745 0 - 0.0803724 2024-01-11T10:57:22.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2397 16 374 358 0.00933356 0 - 0.0803724 2024-01-11T10:57:23.392 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2398 16 374 358 0.00932966 0 - 0.0803724 2024-01-11T10:57:24.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2399 16 374 358 0.00932577 0 - 0.0803724 2024-01-11T10:57:25.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:57:25.393951+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:57:25.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:57:25.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2400 16 374 358 0.00932189 0 - 0.0803724 2024-01-11T10:57:26.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2401 16 374 358 0.00931801 0 - 0.0803724 2024-01-11T10:57:27.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2402 16 374 358 0.00931413 0 - 0.0803724 2024-01-11T10:57:28.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2403 16 374 358 0.00931025 0 - 0.0803724 2024-01-11T10:57:29.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2404 16 374 358 0.00930638 0 - 0.0803724 2024-01-11T10:57:30.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2405 16 374 358 0.00930251 0 - 0.0803724 2024-01-11T10:57:31.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2406 16 374 358 0.00929864 0 - 0.0803724 2024-01-11T10:57:32.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2407 16 374 358 0.00929478 0 - 0.0803724 2024-01-11T10:57:33.393 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2408 16 374 358 0.00929092 0 - 0.0803724 2024-01-11T10:57:34.394 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2409 16 374 358 0.00928706 0 - 0.0803724 2024-01-11T10:57:35.394 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2410 16 374 358 0.00928321 0 - 0.0803724 2024-01-11T10:57:36.394 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2411 16 374 358 0.00927936 0 - 0.0803724 2024-01-11T10:57:37.394 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2412 16 374 358 0.00927551 0 - 0.0803724 2024-01-11T10:57:38.394 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2413 16 374 358 0.00927167 0 - 0.0803724 2024-01-11T10:57:39.394 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2414 16 374 358 0.00926783 0 - 0.0803724 2024-01-11T10:57:40.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2415 16 374 358 0.00926399 0 - 0.0803724 2024-01-11T10:57:41.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2416 16 374 358 0.00926015 0 - 0.0803724 2024-01-11T10:57:42.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2417 16 374 358 0.00925632 0 - 0.0803724 2024-01-11T10:57:43.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2418 16 374 358 0.00925249 0 - 0.0803724 2024-01-11T10:57:44.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2419 16 374 358 0.00924867 0 - 0.0803724 2024-01-11T10:57:45.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:57:45.396490+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:57:45.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:57:45.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2420 16 374 358 0.00924485 0 - 0.0803724 2024-01-11T10:57:46.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2421 16 374 358 0.00924103 0 - 0.0803724 2024-01-11T10:57:47.395 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2422 16 374 358 0.00923721 0 - 0.0803724 2024-01-11T10:57:48.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2423 16 374 358 0.0092334 0 - 0.0803724 2024-01-11T10:57:49.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2424 16 374 358 0.00922959 0 - 0.0803724 2024-01-11T10:57:50.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2425 16 374 358 0.00922578 0 - 0.0803724 2024-01-11T10:57:51.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2426 16 374 358 0.00922198 0 - 0.0803724 2024-01-11T10:57:52.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2427 16 374 358 0.00921818 0 - 0.0803724 2024-01-11T10:57:53.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2428 16 374 358 0.00921439 0 - 0.0803724 2024-01-11T10:57:54.396 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2429 16 374 358 0.00921059 0 - 0.0803724 2024-01-11T10:57:55.397 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2430 16 374 358 0.0092068 0 - 0.0803724 2024-01-11T10:57:56.397 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2431 16 374 358 0.00920301 0 - 0.0803724 2024-01-11T10:57:57.397 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2432 16 374 358 0.00919923 0 - 0.0803724 2024-01-11T10:57:58.397 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2433 16 374 358 0.00919545 0 - 0.0803724 2024-01-11T10:57:59.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2434 16 374 358 0.00919167 0 - 0.0803724 2024-01-11T10:58:00.397 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2435 16 374 358 0.0091879 0 - 0.0803724 2024-01-11T10:58:01.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2436 16 374 358 0.00918412 0 - 0.0803724 2024-01-11T10:58:02.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2437 16 374 358 0.00918035 0 - 0.0803724 2024-01-11T10:58:03.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2438 16 374 358 0.00917659 0 - 0.0803724 2024-01-11T10:58:04.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2439 16 374 358 0.00917283 0 - 0.0803724 2024-01-11T10:58:05.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:58:05.399299+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:58:05.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:58:05.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2440 16 374 358 0.00916907 0 - 0.0803724 2024-01-11T10:58:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T10:58:05.890 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T10:58:06.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2441 16 374 358 0.00916531 0 - 0.0803724 2024-01-11T10:58:07.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2442 16 374 358 0.00916156 0 - 0.0803724 2024-01-11T10:58:07.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T10:58:08.011 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T10:58:08.008+0000 7f502dc24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T10:58:08.398 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2443 16 374 358 0.00915781 0 - 0.0803724 2024-01-11T10:58:09.399 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2444 16 374 358 0.00915406 0 - 0.0803724 2024-01-11T10:58:10.399 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2445 16 374 358 0.00915032 0 - 0.0803724 2024-01-11T10:58:11.399 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2446 16 374 358 0.00914658 0 - 0.0803724 2024-01-11T10:58:12.399 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2447 16 374 358 0.00914284 0 - 0.0803724 2024-01-11T10:58:13.399 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2448 16 374 358 0.0091391 0 - 0.0803724 2024-01-11T10:58:14.399 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2449 16 374 358 0.00913537 0 - 0.0803724 2024-01-11T10:58:15.400 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2450 16 374 358 0.00913164 0 - 0.0803724 2024-01-11T10:58:16.400 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2451 16 374 358 0.00912792 0 - 0.0803724 2024-01-11T10:58:17.400 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2452 16 374 358 0.00912419 0 - 0.0803724 2024-01-11T10:58:18.400 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2453 16 374 358 0.00912047 0 - 0.0803724 2024-01-11T10:58:19.400 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2454 16 374 358 0.00911676 0 - 0.0803724 2024-01-11T10:58:20.400 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2455 16 374 358 0.00911304 0 - 0.0803724 2024-01-11T10:58:21.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2456 16 374 358 0.00910933 0 - 0.0803724 2024-01-11T10:58:22.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2457 16 374 358 0.00910562 0 - 0.0803724 2024-01-11T10:58:23.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2458 16 374 358 0.00910192 0 - 0.0803724 2024-01-11T10:58:24.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2459 16 374 358 0.00909822 0 - 0.0803724 2024-01-11T10:58:25.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:58:25.401999+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:58:25.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:58:25.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2460 16 374 358 0.00909452 0 - 0.0803724 2024-01-11T10:58:26.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2461 16 374 358 0.00909082 0 - 0.0803724 2024-01-11T10:58:27.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2462 16 374 358 0.00908713 0 - 0.0803724 2024-01-11T10:58:28.401 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2463 16 374 358 0.00908344 0 - 0.0803724 2024-01-11T10:58:29.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2464 16 374 358 0.00907976 0 - 0.0803724 2024-01-11T10:58:30.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2465 16 374 358 0.00907607 0 - 0.0803724 2024-01-11T10:58:31.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2466 16 374 358 0.00907239 0 - 0.0803724 2024-01-11T10:58:32.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2467 16 374 358 0.00906871 0 - 0.0803724 2024-01-11T10:58:33.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2468 16 374 358 0.00906504 0 - 0.0803724 2024-01-11T10:58:34.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2469 16 374 358 0.00906137 0 - 0.0803724 2024-01-11T10:58:35.402 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2470 16 374 358 0.0090577 0 - 0.0803724 2024-01-11T10:58:36.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2471 16 374 358 0.00905403 0 - 0.0803724 2024-01-11T10:58:37.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2472 16 374 358 0.00905037 0 - 0.0803724 2024-01-11T10:58:38.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2473 16 374 358 0.00904671 0 - 0.0803724 2024-01-11T10:58:39.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2474 16 374 358 0.00904305 0 - 0.0803724 2024-01-11T10:58:40.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2475 16 374 358 0.0090394 0 - 0.0803724 2024-01-11T10:58:41.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2476 16 374 358 0.00903575 0 - 0.0803724 2024-01-11T10:58:42.403 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2477 16 374 358 0.0090321 0 - 0.0803724 2024-01-11T10:58:43.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2478 16 374 358 0.00902846 0 - 0.0803724 2024-01-11T10:58:44.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2479 16 374 358 0.00902481 0 - 0.0803724 2024-01-11T10:58:45.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:58:45.404760+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:58:45.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:58:45.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2480 16 374 358 0.00902118 0 - 0.0803724 2024-01-11T10:58:46.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2481 16 374 358 0.00901754 0 - 0.0803724 2024-01-11T10:58:47.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2482 16 374 358 0.00901391 0 - 0.0803724 2024-01-11T10:58:48.404 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2483 16 374 358 0.00901028 0 - 0.0803724 2024-01-11T10:58:49.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2484 16 374 358 0.00900665 0 - 0.0803724 2024-01-11T10:58:50.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2485 16 374 358 0.00900302 0 - 0.0803724 2024-01-11T10:58:51.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2486 16 374 358 0.0089994 0 - 0.0803724 2024-01-11T10:58:52.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2487 16 374 358 0.00899578 0 - 0.0803724 2024-01-11T10:58:53.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2488 16 374 358 0.00899217 0 - 0.0803724 2024-01-11T10:58:54.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2489 16 374 358 0.00898855 0 - 0.0803724 2024-01-11T10:58:55.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2490 16 374 358 0.00898494 0 - 0.0803724 2024-01-11T10:58:56.405 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2491 16 374 358 0.00898134 0 - 0.0803724 2024-01-11T10:58:57.406 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2492 16 374 358 0.00897773 0 - 0.0803724 2024-01-11T10:58:58.406 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2493 16 374 358 0.00897413 0 - 0.0803724 2024-01-11T10:58:59.406 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2494 16 374 358 0.00897053 0 - 0.0803724 2024-01-11T10:59:00.406 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2495 16 374 358 0.00896694 0 - 0.0803724 2024-01-11T10:59:01.406 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2496 16 374 358 0.00896335 0 - 0.0803724 2024-01-11T10:59:02.406 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2497 16 374 358 0.00895976 0 - 0.0803724 2024-01-11T10:59:03.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2498 16 374 358 0.00895617 0 - 0.0803724 2024-01-11T10:59:04.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2499 16 374 358 0.00895259 0 - 0.0803724 2024-01-11T10:59:05.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:59:05.407559+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:59:05.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:59:05.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2500 16 374 358 0.008949 0 - 0.0803724 2024-01-11T10:59:06.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2501 16 374 358 0.00894543 0 - 0.0803724 2024-01-11T10:59:07.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2502 16 374 358 0.00894185 0 - 0.0803724 2024-01-11T10:59:08.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2503 16 374 358 0.00893828 0 - 0.0803724 2024-01-11T10:59:09.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2504 16 374 358 0.00893471 0 - 0.0803724 2024-01-11T10:59:10.407 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2505 16 374 358 0.00893114 0 - 0.0803724 2024-01-11T10:59:11.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2506 16 374 358 0.00892758 0 - 0.0803724 2024-01-11T10:59:12.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2507 16 374 358 0.00892402 0 - 0.0803724 2024-01-11T10:59:13.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2508 16 374 358 0.00892046 0 - 0.0803724 2024-01-11T10:59:14.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2509 16 374 358 0.0089169 0 - 0.0803724 2024-01-11T10:59:15.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2510 16 374 358 0.00891335 0 - 0.0803724 2024-01-11T10:59:16.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2511 16 374 358 0.0089098 0 - 0.0803724 2024-01-11T10:59:17.408 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2512 16 374 358 0.00890625 0 - 0.0803724 2024-01-11T10:59:18.409 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2513 16 374 358 0.00890271 0 - 0.0803724 2024-01-11T10:59:19.409 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2514 16 374 358 0.00889917 0 - 0.0803724 2024-01-11T10:59:20.409 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2515 16 374 358 0.00889563 0 - 0.0803724 2024-01-11T10:59:21.409 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2516 16 374 358 0.00889209 0 - 0.0803724 2024-01-11T10:59:22.409 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2517 16 374 358 0.00888856 0 - 0.0803724 2024-01-11T10:59:23.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2518 16 374 358 0.00888503 0 - 0.0803724 2024-01-11T10:59:24.409 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2519 16 374 358 0.0088815 0 - 0.0803724 2024-01-11T10:59:25.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:59:25.410310+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:59:25.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:59:25.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2520 16 374 358 0.00887798 0 - 0.0803724 2024-01-11T10:59:26.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2521 16 374 358 0.00887446 0 - 0.0803724 2024-01-11T10:59:27.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2522 16 374 358 0.00887094 0 - 0.0803724 2024-01-11T10:59:28.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2523 16 374 358 0.00886742 0 - 0.0803724 2024-01-11T10:59:29.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2524 16 374 358 0.00886391 0 - 0.0803724 2024-01-11T10:59:30.410 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2525 16 374 358 0.0088604 0 - 0.0803724 2024-01-11T10:59:31.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2526 16 374 358 0.00885689 0 - 0.0803724 2024-01-11T10:59:32.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2527 16 374 358 0.00885338 0 - 0.0803724 2024-01-11T10:59:33.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2528 16 374 358 0.00884988 0 - 0.0803724 2024-01-11T10:59:34.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2529 16 374 358 0.00884638 0 - 0.0803724 2024-01-11T10:59:35.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2530 16 374 358 0.00884289 0 - 0.0803724 2024-01-11T10:59:36.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2531 16 374 358 0.00883939 0 - 0.0803724 2024-01-11T10:59:37.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2532 16 374 358 0.0088359 0 - 0.0803724 2024-01-11T10:59:38.411 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2533 16 374 358 0.00883241 0 - 0.0803724 2024-01-11T10:59:39.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2534 16 374 358 0.00882893 0 - 0.0803724 2024-01-11T10:59:40.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2535 16 374 358 0.00882544 0 - 0.0803724 2024-01-11T10:59:41.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2536 16 374 358 0.00882196 0 - 0.0803724 2024-01-11T10:59:42.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2537 16 374 358 0.00881849 0 - 0.0803724 2024-01-11T10:59:43.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2538 16 374 358 0.00881501 0 - 0.0803724 2024-01-11T10:59:44.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2539 16 374 358 0.00881154 0 - 0.0803724 2024-01-11T10:59:45.412 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T10:59:45.413110+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T10:59:45.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T10:59:45.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2540 16 374 358 0.00880807 0 - 0.0803724 2024-01-11T10:59:46.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2541 16 374 358 0.0088046 0 - 0.0803724 2024-01-11T10:59:47.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2542 16 374 358 0.00880114 0 - 0.0803724 2024-01-11T10:59:48.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2543 16 374 358 0.00879768 0 - 0.0803724 2024-01-11T10:59:49.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2544 16 374 358 0.00879422 0 - 0.0803724 2024-01-11T10:59:50.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2545 16 374 358 0.00879077 0 - 0.0803724 2024-01-11T10:59:51.413 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2546 16 374 358 0.00878731 0 - 0.0803724 2024-01-11T10:59:52.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2547 16 374 358 0.00878386 0 - 0.0803724 2024-01-11T10:59:53.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2548 16 374 358 0.00878042 0 - 0.0803724 2024-01-11T10:59:54.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2549 16 374 358 0.00877697 0 - 0.0803724 2024-01-11T10:59:55.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2550 16 374 358 0.00877353 0 - 0.0803724 2024-01-11T10:59:56.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2551 16 374 358 0.00877009 0 - 0.0803724 2024-01-11T10:59:57.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2552 16 374 358 0.00876665 0 - 0.0803724 2024-01-11T10:59:58.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2553 16 374 358 0.00876322 0 - 0.0803724 2024-01-11T10:59:59.414 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2554 16 374 358 0.00875979 0 - 0.0803724 2024-01-11T11:00:00.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2555 16 374 358 0.00875636 0 - 0.0803724 2024-01-11T11:00:01.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2556 16 374 358 0.00875293 0 - 0.0803724 2024-01-11T11:00:02.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2557 16 374 358 0.00874951 0 - 0.0803724 2024-01-11T11:00:03.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2558 16 374 358 0.00874609 0 - 0.0803724 2024-01-11T11:00:04.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2559 16 374 358 0.00874267 0 - 0.0803724 2024-01-11T11:00:05.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:00:05.415941+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:00:05.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:00:05.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2560 16 374 358 0.00873926 0 - 0.0803724 2024-01-11T11:00:06.415 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2561 16 374 358 0.00873584 0 - 0.0803724 2024-01-11T11:00:07.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2562 16 374 358 0.00873243 0 - 0.0803724 2024-01-11T11:00:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:00:07.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T11:00:08.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2563 16 374 358 0.00872903 0 - 0.0803724 2024-01-11T11:00:09.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2564 16 374 358 0.00872562 0 - 0.0803724 2024-01-11T11:00:09.930 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T11:00:10.050 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T11:00:10.048+0000 7f9b58c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T11:00:10.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2565 16 374 358 0.00872222 0 - 0.0803724 2024-01-11T11:00:11.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2566 16 374 358 0.00871882 0 - 0.0803724 2024-01-11T11:00:12.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2567 16 374 358 0.00871542 0 - 0.0803724 2024-01-11T11:00:13.416 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2568 16 374 358 0.00871203 0 - 0.0803724 2024-01-11T11:00:14.417 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2569 16 374 358 0.00870864 0 - 0.0803724 2024-01-11T11:00:15.417 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2570 16 374 358 0.00870525 0 - 0.0803724 2024-01-11T11:00:16.417 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2571 16 374 358 0.00870186 0 - 0.0803724 2024-01-11T11:00:17.417 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2572 16 374 358 0.00869848 0 - 0.0803724 2024-01-11T11:00:18.417 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2573 16 374 358 0.0086951 0 - 0.0803724 2024-01-11T11:00:19.417 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2574 16 374 358 0.00869172 0 - 0.0803724 2024-01-11T11:00:20.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2575 16 374 358 0.00868835 0 - 0.0803724 2024-01-11T11:00:21.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2576 16 374 358 0.00868497 0 - 0.0803724 2024-01-11T11:00:22.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2577 16 374 358 0.0086816 0 - 0.0803724 2024-01-11T11:00:23.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2578 16 374 358 0.00867824 0 - 0.0803724 2024-01-11T11:00:24.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2579 16 374 358 0.00867487 0 - 0.0803724 2024-01-11T11:00:25.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:00:25.418692+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:00:25.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:00:25.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2580 16 374 358 0.00867151 0 - 0.0803724 2024-01-11T11:00:26.418 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2581 16 374 358 0.00866815 0 - 0.0803724 2024-01-11T11:00:27.419 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2582 16 374 358 0.00866479 0 - 0.0803724 2024-01-11T11:00:28.419 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2583 16 374 358 0.00866144 0 - 0.0803724 2024-01-11T11:00:29.419 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2584 16 374 358 0.00865808 0 - 0.0803724 2024-01-11T11:00:30.419 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2585 16 374 358 0.00865473 0 - 0.0803724 2024-01-11T11:00:31.419 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2586 16 374 358 0.00865139 0 - 0.0803724 2024-01-11T11:00:32.419 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2587 16 374 358 0.00864804 0 - 0.0803724 2024-01-11T11:00:33.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2588 16 374 358 0.0086447 0 - 0.0803724 2024-01-11T11:00:34.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2589 16 374 358 0.00864136 0 - 0.0803724 2024-01-11T11:00:35.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2590 16 374 358 0.00863803 0 - 0.0803724 2024-01-11T11:00:36.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2591 16 374 358 0.00863469 0 - 0.0803724 2024-01-11T11:00:37.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2592 16 374 358 0.00863136 0 - 0.0803724 2024-01-11T11:00:38.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2593 16 374 358 0.00862803 0 - 0.0803724 2024-01-11T11:00:39.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2594 16 374 358 0.00862471 0 - 0.0803724 2024-01-11T11:00:40.420 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2595 16 374 358 0.00862138 0 - 0.0803724 2024-01-11T11:00:41.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2596 16 374 358 0.00861806 0 - 0.0803724 2024-01-11T11:00:42.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2597 16 374 358 0.00861474 0 - 0.0803724 2024-01-11T11:00:43.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2598 16 374 358 0.00861143 0 - 0.0803724 2024-01-11T11:00:44.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2599 16 374 358 0.00860811 0 - 0.0803724 2024-01-11T11:00:45.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:00:45.421541+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:00:45.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:00:45.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2600 16 374 358 0.0086048 0 - 0.0803724 2024-01-11T11:00:46.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2601 16 374 358 0.00860149 0 - 0.0803724 2024-01-11T11:00:47.421 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2602 16 374 358 0.00859819 0 - 0.0803724 2024-01-11T11:00:48.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2603 16 374 358 0.00859488 0 - 0.0803724 2024-01-11T11:00:49.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2604 16 374 358 0.00859158 0 - 0.0803724 2024-01-11T11:00:50.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2605 16 374 358 0.00858829 0 - 0.0803724 2024-01-11T11:00:51.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2606 16 374 358 0.00858499 0 - 0.0803724 2024-01-11T11:00:52.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2607 16 374 358 0.0085817 0 - 0.0803724 2024-01-11T11:00:53.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2608 16 374 358 0.00857841 0 - 0.0803724 2024-01-11T11:00:54.422 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2609 16 374 358 0.00857512 0 - 0.0803724 2024-01-11T11:00:55.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2610 16 374 358 0.00857183 0 - 0.0803724 2024-01-11T11:00:56.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2611 16 374 358 0.00856855 0 - 0.0803724 2024-01-11T11:00:57.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2612 16 374 358 0.00856527 0 - 0.0803724 2024-01-11T11:00:58.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2613 16 374 358 0.00856199 0 - 0.0803724 2024-01-11T11:00:59.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2614 16 374 358 0.00855872 0 - 0.0803724 2024-01-11T11:01:00.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2615 16 374 358 0.00855544 0 - 0.0803724 2024-01-11T11:01:01.423 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2616 16 374 358 0.00855217 0 - 0.0803724 2024-01-11T11:01:02.424 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2617 16 374 358 0.0085489 0 - 0.0803724 2024-01-11T11:01:03.424 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2618 16 374 358 0.00854564 0 - 0.0803724 2024-01-11T11:01:04.424 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2619 16 374 358 0.00854237 0 - 0.0803724 2024-01-11T11:01:05.424 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:01:05.424624+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:01:05.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:01:05.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2620 16 374 358 0.00853911 0 - 0.0803724 2024-01-11T11:01:06.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2621 16 374 358 0.00853586 0 - 0.0803724 2024-01-11T11:01:07.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2622 16 374 358 0.0085326 0 - 0.0803724 2024-01-11T11:01:08.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2623 16 374 358 0.00852935 0 - 0.0803724 2024-01-11T11:01:09.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2624 16 374 358 0.0085261 0 - 0.0803724 2024-01-11T11:01:10.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2625 16 374 358 0.00852285 0 - 0.0803724 2024-01-11T11:01:11.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2626 16 374 358 0.0085196 0 - 0.0803724 2024-01-11T11:01:12.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2627 16 374 358 0.00851636 0 - 0.0803724 2024-01-11T11:01:13.425 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2628 16 374 358 0.00851312 0 - 0.0803724 2024-01-11T11:01:14.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2629 16 374 358 0.00850988 0 - 0.0803724 2024-01-11T11:01:15.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2630 16 374 358 0.00850664 0 - 0.0803724 2024-01-11T11:01:16.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2631 16 374 358 0.00850341 0 - 0.0803724 2024-01-11T11:01:17.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2632 16 374 358 0.00850018 0 - 0.0803724 2024-01-11T11:01:18.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2633 16 374 358 0.00849695 0 - 0.0803724 2024-01-11T11:01:19.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2634 16 374 358 0.00849373 0 - 0.0803724 2024-01-11T11:01:20.426 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2635 16 374 358 0.0084905 0 - 0.0803724 2024-01-11T11:01:21.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2636 16 374 358 0.00848728 0 - 0.0803724 2024-01-11T11:01:22.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2637 16 374 358 0.00848406 0 - 0.0803724 2024-01-11T11:01:23.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2638 16 374 358 0.00848085 0 - 0.0803724 2024-01-11T11:01:24.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2639 16 374 358 0.00847763 0 - 0.0803724 2024-01-11T11:01:25.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:01:25.427384+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:01:25.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:01:25.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2640 16 374 358 0.00847442 0 - 0.0803724 2024-01-11T11:01:26.427 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2641 16 374 358 0.00847121 0 - 0.0803724 2024-01-11T11:01:27.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2642 16 374 358 0.00846801 0 - 0.0803724 2024-01-11T11:01:28.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2643 16 374 358 0.0084648 0 - 0.0803724 2024-01-11T11:01:29.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2644 16 374 358 0.0084616 0 - 0.0803724 2024-01-11T11:01:30.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2645 16 374 358 0.0084584 0 - 0.0803724 2024-01-11T11:01:31.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2646 16 374 358 0.0084552 0 - 0.0803724 2024-01-11T11:01:32.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2647 16 374 358 0.00845201 0 - 0.0803724 2024-01-11T11:01:33.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2648 16 374 358 0.00844882 0 - 0.0803724 2024-01-11T11:01:34.428 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2649 16 374 358 0.00844563 0 - 0.0803724 2024-01-11T11:01:35.429 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2650 16 374 358 0.00844244 0 - 0.0803724 2024-01-11T11:01:36.429 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2651 16 374 358 0.00843926 0 - 0.0803724 2024-01-11T11:01:37.429 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2652 16 374 358 0.00843607 0 - 0.0803724 2024-01-11T11:01:38.429 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2653 16 374 358 0.00843289 0 - 0.0803724 2024-01-11T11:01:39.429 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2654 16 374 358 0.00842972 0 - 0.0803724 2024-01-11T11:01:40.429 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2655 16 374 358 0.00842654 0 - 0.0803724 2024-01-11T11:01:41.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2656 16 374 358 0.00842337 0 - 0.0803724 2024-01-11T11:01:42.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2657 16 374 358 0.0084202 0 - 0.0803724 2024-01-11T11:01:43.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2658 16 374 358 0.00841703 0 - 0.0803724 2024-01-11T11:01:44.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2659 16 374 358 0.00841387 0 - 0.0803724 2024-01-11T11:01:45.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:01:45.430331+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:01:45.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:01:45.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2660 16 374 358 0.0084107 0 - 0.0803724 2024-01-11T11:01:46.430 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2661 16 374 358 0.00840754 0 - 0.0803724 2024-01-11T11:01:47.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2662 16 374 358 0.00840438 0 - 0.0803724 2024-01-11T11:01:48.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2663 16 374 358 0.00840123 0 - 0.0803724 2024-01-11T11:01:49.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2664 16 374 358 0.00839807 0 - 0.0803724 2024-01-11T11:01:50.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2665 16 374 358 0.00839492 0 - 0.0803724 2024-01-11T11:01:51.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2666 16 374 358 0.00839177 0 - 0.0803724 2024-01-11T11:01:52.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2667 16 374 358 0.00838863 0 - 0.0803724 2024-01-11T11:01:53.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2668 16 374 358 0.00838548 0 - 0.0803724 2024-01-11T11:01:54.431 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2669 16 374 358 0.00838234 0 - 0.0803724 2024-01-11T11:01:55.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2670 16 374 358 0.0083792 0 - 0.0803724 2024-01-11T11:01:56.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2671 16 374 358 0.00837606 0 - 0.0803724 2024-01-11T11:01:57.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2672 16 374 358 0.00837293 0 - 0.0803724 2024-01-11T11:01:58.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2673 16 374 358 0.0083698 0 - 0.0803724 2024-01-11T11:01:59.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2674 16 374 358 0.00836667 0 - 0.0803724 2024-01-11T11:02:00.432 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2675 16 374 358 0.00836354 0 - 0.0803724 2024-01-11T11:02:01.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2676 16 374 358 0.00836041 0 - 0.0803724 2024-01-11T11:02:02.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2677 16 374 358 0.00835729 0 - 0.0803724 2024-01-11T11:02:03.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2678 16 374 358 0.00835417 0 - 0.0803724 2024-01-11T11:02:04.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2679 16 374 358 0.00835105 0 - 0.0803724 2024-01-11T11:02:05.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:02:05.433104+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:02:05.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:02:05.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2680 16 374 358 0.00834793 0 - 0.0803724 2024-01-11T11:02:06.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2681 16 374 358 0.00834482 0 - 0.0803724 2024-01-11T11:02:07.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2682 16 374 358 0.00834171 0 - 0.0803724 2024-01-11T11:02:08.433 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2683 16 374 358 0.0083386 0 - 0.0803724 2024-01-11T11:02:09.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2684 16 374 358 0.00833549 0 - 0.0803724 2024-01-11T11:02:09.969 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:02:09.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T11:02:10.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2685 16 374 358 0.00833239 0 - 0.0803724 2024-01-11T11:02:11.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2686 16 374 358 0.00832929 0 - 0.0803724 2024-01-11T11:02:11.970 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T11:02:12.089 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T11:02:12.087+0000 7f4a7af5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T11:02:12.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2687 16 374 358 0.00832619 0 - 0.0803724 2024-01-11T11:02:13.434 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2688 16 374 358 0.00832309 0 - 0.0803724 2024-01-11T11:02:14.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2689 16 374 358 0.00831999 0 - 0.0803724 2024-01-11T11:02:15.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2690 16 374 358 0.0083169 0 - 0.0803724 2024-01-11T11:02:16.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2691 16 374 358 0.00831381 0 - 0.0803724 2024-01-11T11:02:17.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2692 16 374 358 0.00831072 0 - 0.0803724 2024-01-11T11:02:18.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2693 16 374 358 0.00830763 0 - 0.0803724 2024-01-11T11:02:19.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2694 16 374 358 0.00830455 0 - 0.0803724 2024-01-11T11:02:20.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2695 16 374 358 0.00830147 0 - 0.0803724 2024-01-11T11:02:21.435 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2696 16 374 358 0.00829839 0 - 0.0803724 2024-01-11T11:02:22.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2697 16 374 358 0.00829531 0 - 0.0803724 2024-01-11T11:02:23.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2698 16 374 358 0.00829224 0 - 0.0803724 2024-01-11T11:02:24.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2699 16 374 358 0.00828917 0 - 0.0803724 2024-01-11T11:02:25.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:02:25.436016+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:02:25.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:02:25.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2700 16 374 358 0.0082861 0 - 0.0803724 2024-01-11T11:02:26.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2701 16 374 358 0.00828303 0 - 0.0803724 2024-01-11T11:02:27.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2702 16 374 358 0.00827996 0 - 0.0803724 2024-01-11T11:02:28.436 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2703 16 374 358 0.0082769 0 - 0.0803724 2024-01-11T11:02:29.437 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2704 16 374 358 0.00827384 0 - 0.0803724 2024-01-11T11:02:30.437 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2705 16 374 358 0.00827078 0 - 0.0803724 2024-01-11T11:02:31.437 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2706 16 374 358 0.00826772 0 - 0.0803724 2024-01-11T11:02:32.437 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2707 16 374 358 0.00826467 0 - 0.0803724 2024-01-11T11:02:33.437 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2708 16 374 358 0.00826162 0 - 0.0803724 2024-01-11T11:02:34.437 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2709 16 374 358 0.00825857 0 - 0.0803724 2024-01-11T11:02:35.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2710 16 374 358 0.00825552 0 - 0.0803724 2024-01-11T11:02:36.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2711 16 374 358 0.00825247 0 - 0.0803724 2024-01-11T11:02:37.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2712 16 374 358 0.00824943 0 - 0.0803724 2024-01-11T11:02:38.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2713 16 374 358 0.00824639 0 - 0.0803724 2024-01-11T11:02:39.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2714 16 374 358 0.00824335 0 - 0.0803724 2024-01-11T11:02:40.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2715 16 374 358 0.00824032 0 - 0.0803724 2024-01-11T11:02:41.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2716 16 374 358 0.00823728 0 - 0.0803724 2024-01-11T11:02:42.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2717 16 374 358 0.00823425 0 - 0.0803724 2024-01-11T11:02:43.438 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2718 16 374 358 0.00823122 0 - 0.0803724 2024-01-11T11:02:44.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2719 16 374 358 0.00822819 0 - 0.0803724 2024-01-11T11:02:45.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:02:45.438739+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:02:45.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:02:45.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2720 16 374 358 0.00822517 0 - 0.0803724 2024-01-11T11:02:46.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2721 16 374 358 0.00822214 0 - 0.0803724 2024-01-11T11:02:47.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2722 16 374 358 0.00821912 0 - 0.0803724 2024-01-11T11:02:48.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2723 16 374 358 0.00821611 0 - 0.0803724 2024-01-11T11:02:49.439 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2724 16 374 358 0.00821309 0 - 0.0803724 2024-01-11T11:02:50.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2725 16 374 358 0.00821007 0 - 0.0803724 2024-01-11T11:02:51.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2726 16 374 358 0.00820706 0 - 0.0803724 2024-01-11T11:02:52.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2727 16 374 358 0.00820405 0 - 0.0803724 2024-01-11T11:02:53.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2728 16 374 358 0.00820105 0 - 0.0803724 2024-01-11T11:02:54.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2729 16 374 358 0.00819804 0 - 0.0803724 2024-01-11T11:02:55.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2730 16 374 358 0.00819504 0 - 0.0803724 2024-01-11T11:02:56.440 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2731 16 374 358 0.00819204 0 - 0.0803724 2024-01-11T11:02:57.441 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2732 16 374 358 0.00818904 0 - 0.0803724 2024-01-11T11:02:58.441 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2733 16 374 358 0.00818604 0 - 0.0803724 2024-01-11T11:02:59.441 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2734 16 374 358 0.00818305 0 - 0.0803724 2024-01-11T11:03:00.441 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2735 16 374 358 0.00818006 0 - 0.0803724 2024-01-11T11:03:01.441 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2736 16 374 358 0.00817707 0 - 0.0803724 2024-01-11T11:03:02.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2737 16 374 358 0.00817408 0 - 0.0803724 2024-01-11T11:03:03.441 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2738 16 374 358 0.00817109 0 - 0.0803724 2024-01-11T11:03:04.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2739 16 374 358 0.00816811 0 - 0.0803724 2024-01-11T11:03:05.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:03:05.441494+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:03:05.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:03:05.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2740 16 374 358 0.00816513 0 - 0.0803724 2024-01-11T11:03:06.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2741 16 374 358 0.00816215 0 - 0.0803724 2024-01-11T11:03:07.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2742 16 374 358 0.00815917 0 - 0.0803724 2024-01-11T11:03:08.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2743 16 374 358 0.0081562 0 - 0.0803724 2024-01-11T11:03:09.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2744 16 374 358 0.00815323 0 - 0.0803724 2024-01-11T11:03:10.442 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2745 16 374 358 0.00815026 0 - 0.0803724 2024-01-11T11:03:11.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2746 16 374 358 0.00814729 0 - 0.0803724 2024-01-11T11:03:12.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2747 16 374 358 0.00814432 0 - 0.0803724 2024-01-11T11:03:13.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2748 16 374 358 0.00814136 0 - 0.0803724 2024-01-11T11:03:14.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2749 16 374 358 0.0081384 0 - 0.0803724 2024-01-11T11:03:15.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2750 16 374 358 0.00813544 0 - 0.0803724 2024-01-11T11:03:16.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2751 16 374 358 0.00813248 0 - 0.0803724 2024-01-11T11:03:17.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2752 16 374 358 0.00812952 0 - 0.0803724 2024-01-11T11:03:18.443 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2753 16 374 358 0.00812657 0 - 0.0803724 2024-01-11T11:03:19.444 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2754 16 374 358 0.00812362 0 - 0.0803724 2024-01-11T11:03:20.444 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2755 16 374 358 0.00812067 0 - 0.0803724 2024-01-11T11:03:21.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2756 16 374 358 0.00811772 0 - 0.0803724 2024-01-11T11:03:22.444 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2757 16 374 358 0.00811478 0 - 0.0803724 2024-01-11T11:03:23.444 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2758 16 374 358 0.00811184 0 - 0.0803724 2024-01-11T11:03:24.444 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2759 16 374 358 0.0081089 0 - 0.0803724 2024-01-11T11:03:25.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:03:25.444249+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:03:25.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:03:25.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2760 16 374 358 0.00810596 0 - 0.0803724 2024-01-11T11:03:26.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2761 16 374 358 0.00810302 0 - 0.0803724 2024-01-11T11:03:27.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2762 16 374 358 0.00810009 0 - 0.0803724 2024-01-11T11:03:28.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2763 16 374 358 0.00809716 0 - 0.0803724 2024-01-11T11:03:29.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2764 16 374 358 0.00809423 0 - 0.0803724 2024-01-11T11:03:30.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2765 16 374 358 0.0080913 0 - 0.0803724 2024-01-11T11:03:31.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2766 16 374 358 0.00808838 0 - 0.0803724 2024-01-11T11:03:32.445 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2767 16 374 358 0.00808545 0 - 0.0803724 2024-01-11T11:03:33.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2768 16 374 358 0.00808253 0 - 0.0803724 2024-01-11T11:03:34.446 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2769 16 374 358 0.00807961 0 - 0.0803724 2024-01-11T11:03:35.446 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2770 16 374 358 0.0080767 0 - 0.0803724 2024-01-11T11:03:36.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2771 16 374 358 0.00807378 0 - 0.0803724 2024-01-11T11:03:37.446 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2772 16 374 358 0.00807087 0 - 0.0803724 2024-01-11T11:03:38.446 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2773 16 374 358 0.00806796 0 - 0.0803724 2024-01-11T11:03:39.446 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2774 16 374 358 0.00806505 0 - 0.0803724 2024-01-11T11:03:40.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2775 16 374 358 0.00806214 0 - 0.0803724 2024-01-11T11:03:41.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2776 16 374 358 0.00805924 0 - 0.0803724 2024-01-11T11:03:42.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2777 16 374 358 0.00805634 0 - 0.0803724 2024-01-11T11:03:43.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2778 16 374 358 0.00805344 0 - 0.0803724 2024-01-11T11:03:44.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2779 16 374 358 0.00805054 0 - 0.0803724 2024-01-11T11:03:45.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:03:45.446733+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:03:45.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:03:45.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2780 16 374 358 0.00804764 0 - 0.0803724 2024-01-11T11:03:46.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2781 16 374 358 0.00804475 0 - 0.0803724 2024-01-11T11:03:47.447 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2782 16 374 358 0.00804186 0 - 0.0803724 2024-01-11T11:03:48.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2783 16 374 358 0.00803897 0 - 0.0803724 2024-01-11T11:03:49.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2784 16 374 358 0.00803608 0 - 0.0803724 2024-01-11T11:03:50.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2785 16 374 358 0.00803319 0 - 0.0803724 2024-01-11T11:03:51.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2786 16 374 358 0.00803031 0 - 0.0803724 2024-01-11T11:03:52.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2787 16 374 358 0.00802743 0 - 0.0803724 2024-01-11T11:03:53.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2788 16 374 358 0.00802455 0 - 0.0803724 2024-01-11T11:03:54.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2789 16 374 358 0.00802167 0 - 0.0803724 2024-01-11T11:03:55.448 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2790 16 374 358 0.0080188 0 - 0.0803724 2024-01-11T11:03:56.449 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2791 16 374 358 0.00801592 0 - 0.0803724 2024-01-11T11:03:57.449 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2792 16 374 358 0.00801305 0 - 0.0803724 2024-01-11T11:03:58.449 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2793 16 374 358 0.00801018 0 - 0.0803724 2024-01-11T11:03:59.449 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2794 16 374 358 0.00800732 0 - 0.0803724 2024-01-11T11:04:00.449 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2795 16 374 358 0.00800445 0 - 0.0803724 2024-01-11T11:04:01.449 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2796 16 374 358 0.00800159 0 - 0.0803724 2024-01-11T11:04:02.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2797 16 374 358 0.00799873 0 - 0.0803724 2024-01-11T11:04:03.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2798 16 374 358 0.00799587 0 - 0.0803724 2024-01-11T11:04:04.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2799 16 374 358 0.00799301 0 - 0.0803724 2024-01-11T11:04:05.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:04:05.449639+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:04:05.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:04:05.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2800 16 374 358 0.00799016 0 - 0.0803724 2024-01-11T11:04:06.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2801 16 374 358 0.0079873 0 - 0.0803724 2024-01-11T11:04:07.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2802 16 374 358 0.00798445 0 - 0.0803724 2024-01-11T11:04:08.450 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2803 16 374 358 0.00798161 0 - 0.0803724 2024-01-11T11:04:09.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2804 16 374 358 0.00797876 0 - 0.0803724 2024-01-11T11:04:10.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2805 16 374 358 0.00797591 0 - 0.0803724 2024-01-11T11:04:11.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2806 16 374 358 0.00797307 0 - 0.0803724 2024-01-11T11:04:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:04:12.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T11:04:12.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2807 16 374 358 0.00797023 0 - 0.0803724 2024-01-11T11:04:13.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2808 16 374 358 0.00796739 0 - 0.0803724 2024-01-11T11:04:14.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-01-11T11:04:14.130 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T11:04:14.126+0000 7f1643b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T11:04:14.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2809 16 374 358 0.00796456 0 - 0.0803724 2024-01-11T11:04:15.451 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2810 16 374 358 0.00796172 0 - 0.0803724 2024-01-11T11:04:16.452 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2811 16 374 358 0.00795889 0 - 0.0803724 2024-01-11T11:04:17.452 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2812 16 374 358 0.00795606 0 - 0.0803724 2024-01-11T11:04:18.452 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2813 16 374 358 0.00795323 0 - 0.0803724 2024-01-11T11:04:19.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2814 16 374 358 0.0079504 0 - 0.0803724 2024-01-11T11:04:20.452 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2815 16 374 358 0.00794758 0 - 0.0803724 2024-01-11T11:04:21.452 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2816 16 374 358 0.00794476 0 - 0.0803724 2024-01-11T11:04:22.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2817 16 374 358 0.00794194 0 - 0.0803724 2024-01-11T11:04:23.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2818 16 374 358 0.00793912 0 - 0.0803724 2024-01-11T11:04:24.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2819 16 374 358 0.0079363 0 - 0.0803724 2024-01-11T11:04:25.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:04:25.452505+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:04:25.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:04:25.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2820 16 374 358 0.00793349 0 - 0.0803724 2024-01-11T11:04:26.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2821 16 374 358 0.00793068 0 - 0.0803724 2024-01-11T11:04:27.453 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2822 16 374 358 0.00792787 0 - 0.0803724 2024-01-11T11:04:28.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2823 16 374 358 0.00792506 0 - 0.0803724 2024-01-11T11:04:29.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2824 16 374 358 0.00792225 0 - 0.0803724 2024-01-11T11:04:30.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2825 16 374 358 0.00791945 0 - 0.0803724 2024-01-11T11:04:31.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2826 16 374 358 0.00791664 0 - 0.0803724 2024-01-11T11:04:32.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2827 16 374 358 0.00791384 0 - 0.0803724 2024-01-11T11:04:33.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2828 16 374 358 0.00791104 0 - 0.0803724 2024-01-11T11:04:34.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2829 16 374 358 0.00790825 0 - 0.0803724 2024-01-11T11:04:35.454 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2830 16 374 358 0.00790545 0 - 0.0803724 2024-01-11T11:04:36.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2831 16 374 358 0.00790266 0 - 0.0803724 2024-01-11T11:04:37.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2832 16 374 358 0.00789987 0 - 0.0803724 2024-01-11T11:04:38.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2833 16 374 358 0.00789708 0 - 0.0803724 2024-01-11T11:04:39.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2834 16 374 358 0.0078943 0 - 0.0803724 2024-01-11T11:04:40.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2835 16 374 358 0.00789151 0 - 0.0803724 2024-01-11T11:04:41.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2836 16 374 358 0.00788873 0 - 0.0803724 2024-01-11T11:04:42.455 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2837 16 374 358 0.00788595 0 - 0.0803724 2024-01-11T11:04:43.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2838 16 374 358 0.00788317 0 - 0.0803724 2024-01-11T11:04:44.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2839 16 374 358 0.00788039 0 - 0.0803724 2024-01-11T11:04:45.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:04:45.455355+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:04:45.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:04:45.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2840 16 374 358 0.00787762 0 - 0.0803724 2024-01-11T11:04:46.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2841 16 374 358 0.00787484 0 - 0.0803724 2024-01-11T11:04:47.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2842 16 374 358 0.00787207 0 - 0.0803724 2024-01-11T11:04:48.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2843 16 374 358 0.0078693 0 - 0.0803724 2024-01-11T11:04:49.456 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2844 16 374 358 0.00786654 0 - 0.0803724 2024-01-11T11:04:50.457 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2845 16 374 358 0.00786377 0 - 0.0803724 2024-01-11T11:04:51.457 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2846 16 374 358 0.00786101 0 - 0.0803724 2024-01-11T11:04:52.457 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2847 16 374 358 0.00785825 0 - 0.0803724 2024-01-11T11:04:53.457 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2848 16 374 358 0.00785549 0 - 0.0803724 2024-01-11T11:04:54.457 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2849 16 374 358 0.00785273 0 - 0.0803724 2024-01-11T11:04:55.457 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2850 16 374 358 0.00784997 0 - 0.0803724 2024-01-11T11:04:56.458 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2851 16 374 358 0.00784722 0 - 0.0803724 2024-01-11T11:04:57.458 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2852 16 374 358 0.00784447 0 - 0.0803724 2024-01-11T11:04:58.458 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2853 16 374 358 0.00784172 0 - 0.0803724 2024-01-11T11:04:59.458 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2854 16 374 358 0.00783897 0 - 0.0803724 2024-01-11T11:05:00.458 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2855 16 374 358 0.00783623 0 - 0.0803724 2024-01-11T11:05:01.458 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2856 16 374 358 0.00783348 0 - 0.0803724 2024-01-11T11:05:02.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2857 16 374 358 0.00783074 0 - 0.0803724 2024-01-11T11:05:03.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2858 16 374 358 0.007828 0 - 0.0803724 2024-01-11T11:05:04.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2859 16 374 358 0.00782526 0 - 0.0803724 2024-01-11T11:05:05.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:05:05.458247+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:05:05.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:05:05.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2860 16 374 358 0.00782253 0 - 0.0803724 2024-01-11T11:05:06.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2861 16 374 358 0.00781979 0 - 0.0803724 2024-01-11T11:05:07.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2862 16 374 358 0.00781706 0 - 0.0803724 2024-01-11T11:05:08.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2863 16 374 358 0.00781433 0 - 0.0803724 2024-01-11T11:05:09.459 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2864 16 374 358 0.0078116 0 - 0.0803724 2024-01-11T11:05:10.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2865 16 374 358 0.00780887 0 - 0.0803724 2024-01-11T11:05:11.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2866 16 374 358 0.00780615 0 - 0.0803724 2024-01-11T11:05:12.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2867 16 374 358 0.00780343 0 - 0.0803724 2024-01-11T11:05:13.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2868 16 374 358 0.00780071 0 - 0.0803724 2024-01-11T11:05:14.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2869 16 374 358 0.00779799 0 - 0.0803724 2024-01-11T11:05:15.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2870 16 374 358 0.00779527 0 - 0.0803724 2024-01-11T11:05:16.460 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2871 16 374 358 0.00779255 0 - 0.0803724 2024-01-11T11:05:17.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2872 16 374 358 0.00778984 0 - 0.0803724 2024-01-11T11:05:18.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2873 16 374 358 0.00778713 0 - 0.0803724 2024-01-11T11:05:19.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2874 16 374 358 0.00778442 0 - 0.0803724 2024-01-11T11:05:20.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2875 16 374 358 0.00778171 0 - 0.0803724 2024-01-11T11:05:21.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2876 16 374 358 0.00777901 0 - 0.0803724 2024-01-11T11:05:22.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2877 16 374 358 0.0077763 0 - 0.0803724 2024-01-11T11:05:23.461 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2878 16 374 358 0.0077736 0 - 0.0803724 2024-01-11T11:05:24.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2879 16 374 358 0.0077709 0 - 0.0803724 2024-01-11T11:05:25.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:05:25.461053+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:05:25.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:05:25.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2880 16 374 358 0.0077682 0 - 0.0803724 2024-01-11T11:05:26.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2881 16 374 358 0.00776551 0 - 0.0803724 2024-01-11T11:05:27.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2882 16 374 358 0.00776281 0 - 0.0803724 2024-01-11T11:05:28.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2883 16 374 358 0.00776012 0 - 0.0803724 2024-01-11T11:05:29.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2884 16 374 358 0.00775743 0 - 0.0803724 2024-01-11T11:05:30.462 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2885 16 374 358 0.00775474 0 - 0.0803724 2024-01-11T11:05:31.463 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2886 16 374 358 0.00775205 0 - 0.0803724 2024-01-11T11:05:32.463 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2887 16 374 358 0.00774937 0 - 0.0803724 2024-01-11T11:05:33.463 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2888 16 374 358 0.00774668 0 - 0.0803724 2024-01-11T11:05:34.463 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2889 16 374 358 0.007744 0 - 0.0803724 2024-01-11T11:05:35.463 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2890 16 374 358 0.00774132 0 - 0.0803724 2024-01-11T11:05:36.463 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2891 16 374 358 0.00773864 0 - 0.0803724 2024-01-11T11:05:37.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2892 16 374 358 0.00773597 0 - 0.0803724 2024-01-11T11:05:38.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2893 16 374 358 0.00773329 0 - 0.0803724 2024-01-11T11:05:39.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2894 16 374 358 0.00773062 0 - 0.0803724 2024-01-11T11:05:40.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2895 16 374 358 0.00772795 0 - 0.0803724 2024-01-11T11:05:41.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2896 16 374 358 0.00772528 0 - 0.0803724 2024-01-11T11:05:42.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2897 16 374 358 0.00772262 0 - 0.0803724 2024-01-11T11:05:43.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2898 16 374 358 0.00771995 0 - 0.0803724 2024-01-11T11:05:44.464 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2899 16 374 358 0.00771729 0 - 0.0803724 2024-01-11T11:05:45.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:05:45.463844+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:05:45.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:05:45.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2900 16 374 358 0.00771463 0 - 0.0803724 2024-01-11T11:05:46.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2901 16 374 358 0.00771197 0 - 0.0803724 2024-01-11T11:05:47.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2902 16 374 358 0.00770931 0 - 0.0803724 2024-01-11T11:05:48.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2903 16 374 358 0.00770665 0 - 0.0803724 2024-01-11T11:05:49.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2904 16 374 358 0.007704 0 - 0.0803724 2024-01-11T11:05:50.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2905 16 374 358 0.00770135 0 - 0.0803724 2024-01-11T11:05:51.465 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2906 16 374 358 0.0076987 0 - 0.0803724 2024-01-11T11:05:52.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2907 16 374 358 0.00769605 0 - 0.0803724 2024-01-11T11:05:53.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2908 16 374 358 0.0076934 0 - 0.0803724 2024-01-11T11:05:54.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2909 16 374 358 0.00769076 0 - 0.0803724 2024-01-11T11:05:55.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2910 16 374 358 0.00768812 0 - 0.0803724 2024-01-11T11:05:56.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2911 16 374 358 0.00768547 0 - 0.0803724 2024-01-11T11:05:57.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2912 16 374 358 0.00768284 0 - 0.0803724 2024-01-11T11:05:58.466 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2913 16 374 358 0.0076802 0 - 0.0803724 2024-01-11T11:05:59.467 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2914 16 374 358 0.00767756 0 - 0.0803724 2024-01-11T11:06:00.467 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2915 16 374 358 0.00767493 0 - 0.0803724 2024-01-11T11:06:01.467 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2916 16 374 358 0.0076723 0 - 0.0803724 2024-01-11T11:06:02.467 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2917 16 374 358 0.00766967 0 - 0.0803724 2024-01-11T11:06:03.467 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2918 16 374 358 0.00766704 0 - 0.0803724 2024-01-11T11:06:04.467 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2919 16 374 358 0.00766441 0 - 0.0803724 2024-01-11T11:06:05.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:06:05.466754+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:06:05.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:06:05.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2920 16 374 358 0.00766179 0 - 0.0803724 2024-01-11T11:06:06.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2921 16 374 358 0.00765916 0 - 0.0803724 2024-01-11T11:06:07.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2922 16 374 358 0.00765654 0 - 0.0803724 2024-01-11T11:06:08.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2923 16 374 358 0.00765392 0 - 0.0803724 2024-01-11T11:06:09.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2924 16 374 358 0.0076513 0 - 0.0803724 2024-01-11T11:06:10.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2925 16 374 358 0.00764869 0 - 0.0803724 2024-01-11T11:06:11.468 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2926 16 374 358 0.00764607 0 - 0.0803724 2024-01-11T11:06:12.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2927 16 374 358 0.00764346 0 - 0.0803724 2024-01-11T11:06:13.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2928 16 374 358 0.00764085 0 - 0.0803724 2024-01-11T11:06:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:06:14.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T11:06:14.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2929 16 374 358 0.00763824 0 - 0.0803724 2024-01-11T11:06:15.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2930 16 374 358 0.00763564 0 - 0.0803724 2024-01-11T11:06:16.052 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-01-11T11:06:16.172 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T11:06:16.168+0000 7f24390ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T11:06:16.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2931 16 374 358 0.00763303 0 - 0.0803724 2024-01-11T11:06:17.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2932 16 374 358 0.00763043 0 - 0.0803724 2024-01-11T11:06:18.470 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2933 16 374 358 0.00762783 0 - 0.0803724 2024-01-11T11:06:19.469 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2934 16 374 358 0.00762523 0 - 0.0803724 2024-01-11T11:06:20.470 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2935 16 374 358 0.00762263 0 - 0.0803724 2024-01-11T11:06:21.470 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2936 16 374 358 0.00762003 0 - 0.0803724 2024-01-11T11:06:22.470 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2937 16 374 358 0.00761744 0 - 0.0803724 2024-01-11T11:06:23.470 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2938 16 374 358 0.00761484 0 - 0.0803724 2024-01-11T11:06:24.470 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2939 16 374 358 0.00761225 0 - 0.0803724 2024-01-11T11:06:25.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:06:25.469610+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:06:25.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:06:25.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2940 16 374 358 0.00760966 0 - 0.0803724 2024-01-11T11:06:26.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2941 16 374 358 0.00760708 0 - 0.0803724 2024-01-11T11:06:27.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2942 16 374 358 0.00760449 0 - 0.0803724 2024-01-11T11:06:28.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2943 16 374 358 0.00760191 0 - 0.0803724 2024-01-11T11:06:29.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2944 16 374 358 0.00759932 0 - 0.0803724 2024-01-11T11:06:30.471 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2945 16 374 358 0.00759674 0 - 0.0803724 2024-01-11T11:06:31.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2946 16 374 358 0.00759416 0 - 0.0803724 2024-01-11T11:06:32.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2947 16 374 358 0.00759159 0 - 0.0803724 2024-01-11T11:06:33.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2948 16 374 358 0.00758901 0 - 0.0803724 2024-01-11T11:06:34.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2949 16 374 358 0.00758644 0 - 0.0803724 2024-01-11T11:06:35.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2950 16 374 358 0.00758387 0 - 0.0803724 2024-01-11T11:06:36.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2951 16 374 358 0.0075813 0 - 0.0803724 2024-01-11T11:06:37.472 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2952 16 374 358 0.00757873 0 - 0.0803724 2024-01-11T11:06:38.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2953 16 374 358 0.00757616 0 - 0.0803724 2024-01-11T11:06:39.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2954 16 374 358 0.0075736 0 - 0.0803724 2024-01-11T11:06:40.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2955 16 374 358 0.00757103 0 - 0.0803724 2024-01-11T11:06:41.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2956 16 374 358 0.00756847 0 - 0.0803724 2024-01-11T11:06:42.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2957 16 374 358 0.00756591 0 - 0.0803724 2024-01-11T11:06:43.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2958 16 374 358 0.00756336 0 - 0.0803724 2024-01-11T11:06:44.473 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2959 16 374 358 0.0075608 0 - 0.0803724 2024-01-11T11:06:45.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:06:45.472505+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:06:45.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:06:45.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2960 16 374 358 0.00755825 0 - 0.0803724 2024-01-11T11:06:46.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2961 16 374 358 0.00755569 0 - 0.0803724 2024-01-11T11:06:47.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2962 16 374 358 0.00755314 0 - 0.0803724 2024-01-11T11:06:48.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2963 16 374 358 0.00755059 0 - 0.0803724 2024-01-11T11:06:49.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2964 16 374 358 0.00754805 0 - 0.0803724 2024-01-11T11:06:50.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2965 16 374 358 0.0075455 0 - 0.0803724 2024-01-11T11:06:51.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2966 16 374 358 0.00754296 0 - 0.0803724 2024-01-11T11:06:52.474 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2967 16 374 358 0.00754041 0 - 0.0803724 2024-01-11T11:06:53.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2968 16 374 358 0.00753787 0 - 0.0803724 2024-01-11T11:06:54.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2969 16 374 358 0.00753533 0 - 0.0803724 2024-01-11T11:06:55.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2970 16 374 358 0.0075328 0 - 0.0803724 2024-01-11T11:06:56.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2971 16 374 358 0.00753026 0 - 0.0803724 2024-01-11T11:06:57.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2972 16 374 358 0.00752773 0 - 0.0803724 2024-01-11T11:06:58.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2973 16 374 358 0.00752519 0 - 0.0803724 2024-01-11T11:06:59.475 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2974 16 374 358 0.00752266 0 - 0.0803724 2024-01-11T11:07:00.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2975 16 374 358 0.00752014 0 - 0.0803724 2024-01-11T11:07:01.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2976 16 374 358 0.00751761 0 - 0.0803724 2024-01-11T11:07:02.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2977 16 374 358 0.00751508 0 - 0.0803724 2024-01-11T11:07:03.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2978 16 374 358 0.00751256 0 - 0.0803724 2024-01-11T11:07:04.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2979 16 374 358 0.00751004 0 - 0.0803724 2024-01-11T11:07:05.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:07:05.475227+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:07:05.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:07:05.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2980 16 374 358 0.00750752 0 - 0.0803724 2024-01-11T11:07:06.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2981 16 374 358 0.007505 0 - 0.0803724 2024-01-11T11:07:07.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2982 16 374 358 0.00750248 0 - 0.0803724 2024-01-11T11:07:08.476 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2983 16 374 358 0.00749997 0 - 0.0803724 2024-01-11T11:07:09.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2984 16 374 358 0.00749745 0 - 0.0803724 2024-01-11T11:07:10.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2985 16 374 358 0.00749494 0 - 0.0803724 2024-01-11T11:07:11.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2986 16 374 358 0.00749243 0 - 0.0803724 2024-01-11T11:07:12.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2987 16 374 358 0.00748992 0 - 0.0803724 2024-01-11T11:07:13.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2988 16 374 358 0.00748742 0 - 0.0803724 2024-01-11T11:07:14.477 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2989 16 374 358 0.00748491 0 - 0.0803724 2024-01-11T11:07:15.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2990 16 374 358 0.00748241 0 - 0.0803724 2024-01-11T11:07:16.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2991 16 374 358 0.00747991 0 - 0.0803724 2024-01-11T11:07:17.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2992 16 374 358 0.00747741 0 - 0.0803724 2024-01-11T11:07:18.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2993 16 374 358 0.00747491 0 - 0.0803724 2024-01-11T11:07:19.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2994 16 374 358 0.00747241 0 - 0.0803724 2024-01-11T11:07:20.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2995 16 374 358 0.00746992 0 - 0.0803724 2024-01-11T11:07:21.478 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2996 16 374 358 0.00746742 0 - 0.0803724 2024-01-11T11:07:22.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2997 16 374 358 0.00746493 0 - 0.0803724 2024-01-11T11:07:23.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2998 16 374 358 0.00746244 0 - 0.0803724 2024-01-11T11:07:24.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 2999 16 374 358 0.00745995 0 - 0.0803724 2024-01-11T11:07:25.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:07:25.477944+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:07:25.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:07:25.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3000 16 374 358 0.00745747 0 - 0.0803724 2024-01-11T11:07:26.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3001 16 374 358 0.00745498 0 - 0.0803724 2024-01-11T11:07:27.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3002 16 374 358 0.0074525 0 - 0.0803724 2024-01-11T11:07:28.479 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3003 16 374 358 0.00745002 0 - 0.0803724 2024-01-11T11:07:29.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3004 16 374 358 0.00744754 0 - 0.0803724 2024-01-11T11:07:30.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3005 16 374 358 0.00744506 0 - 0.0803724 2024-01-11T11:07:31.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3006 16 374 358 0.00744258 0 - 0.0803724 2024-01-11T11:07:32.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3007 16 374 358 0.00744011 0 - 0.0803724 2024-01-11T11:07:33.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3008 16 374 358 0.00743763 0 - 0.0803724 2024-01-11T11:07:34.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3009 16 374 358 0.00743516 0 - 0.0803724 2024-01-11T11:07:35.480 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3010 16 374 358 0.00743269 0 - 0.0803724 2024-01-11T11:07:36.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3011 16 374 358 0.00743022 0 - 0.0803724 2024-01-11T11:07:37.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3012 16 374 358 0.00742775 0 - 0.0803724 2024-01-11T11:07:38.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3013 16 374 358 0.00742529 0 - 0.0803724 2024-01-11T11:07:39.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3014 16 374 358 0.00742283 0 - 0.0803724 2024-01-11T11:07:40.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3015 16 374 358 0.00742036 0 - 0.0803724 2024-01-11T11:07:41.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3016 16 374 358 0.0074179 0 - 0.0803724 2024-01-11T11:07:42.481 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3017 16 374 358 0.00741544 0 - 0.0803724 2024-01-11T11:07:43.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3018 16 374 358 0.00741299 0 - 0.0803724 2024-01-11T11:07:44.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3019 16 374 358 0.00741053 0 - 0.0803724 2024-01-11T11:07:45.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:07:45.480796+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:07:45.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:07:45.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3020 16 374 358 0.00740808 0 - 0.0803724 2024-01-11T11:07:46.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3021 16 374 358 0.00740563 0 - 0.0803724 2024-01-11T11:07:47.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3022 16 374 358 0.00740318 0 - 0.0803724 2024-01-11T11:07:48.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3023 16 374 358 0.00740073 0 - 0.0803724 2024-01-11T11:07:49.482 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3024 16 374 358 0.00739828 0 - 0.0803724 2024-01-11T11:07:50.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3025 16 374 358 0.00739583 0 - 0.0803724 2024-01-11T11:07:51.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3026 16 374 358 0.00739339 0 - 0.0803724 2024-01-11T11:07:52.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3027 16 374 358 0.00739095 0 - 0.0803724 2024-01-11T11:07:53.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3028 16 374 358 0.00738851 0 - 0.0803724 2024-01-11T11:07:54.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3029 16 374 358 0.00738607 0 - 0.0803724 2024-01-11T11:07:55.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3030 16 374 358 0.00738363 0 - 0.0803724 2024-01-11T11:07:56.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3031 16 374 358 0.00738119 0 - 0.0803724 2024-01-11T11:07:57.483 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3032 16 374 358 0.00737876 0 - 0.0803724 2024-01-11T11:07:58.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3033 16 374 358 0.00737633 0 - 0.0803724 2024-01-11T11:07:59.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3034 16 374 358 0.00737389 0 - 0.0803724 2024-01-11T11:08:00.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3035 16 374 358 0.00737146 0 - 0.0803724 2024-01-11T11:08:01.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3036 16 374 358 0.00736904 0 - 0.0803724 2024-01-11T11:08:02.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3037 16 374 358 0.00736661 0 - 0.0803724 2024-01-11T11:08:03.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3038 16 374 358 0.00736418 0 - 0.0803724 2024-01-11T11:08:04.484 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3039 16 374 358 0.00736176 0 - 0.0803724 2024-01-11T11:08:05.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:08:05.483500+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:08:05.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:08:05.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3040 16 374 358 0.00735934 0 - 0.0803724 2024-01-11T11:08:06.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3041 16 374 358 0.00735692 0 - 0.0803724 2024-01-11T11:08:07.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3042 16 374 358 0.0073545 0 - 0.0803724 2024-01-11T11:08:08.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3043 16 374 358 0.00735208 0 - 0.0803724 2024-01-11T11:08:09.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3044 16 374 358 0.00734967 0 - 0.0803724 2024-01-11T11:08:09.754 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-01-11T11:08:09.952 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0832196059b14ea697e5435449452a61 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/full_sequential.py", line 37, in task mgr.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/radosbench.py", line 141, in task run.wait(radosbench.values(), timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (501) after waiting for 3000 seconds 2024-01-11T11:08:09.954 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-01-11T11:08:09.964 INFO:tasks.thrashosds:joining thrashosds 2024-01-11T11:08:09.964 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-01-11T11:08:10.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3045 16 374 358 0.00734726 0 - 0.0803724 2024-01-11T11:08:11.485 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3046 16 374 358 0.00734484 0 - 0.0803724 2024-01-11T11:08:12.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3047 16 374 358 0.00734243 0 - 0.0803724 2024-01-11T11:08:13.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3048 16 374 358 0.00734002 0 - 0.0803724 2024-01-11T11:08:14.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3049 16 374 358 0.00733762 0 - 0.0803724 2024-01-11T11:08:15.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3050 16 374 358 0.00733521 0 - 0.0803724 2024-01-11T11:08:16.090 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:08:16.091 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-01-11T11:08:16.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3051 16 374 358 0.00733281 0 - 0.0803724 2024-01-11T11:08:17.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3052 16 374 358 0.0073304 0 - 0.0803724 2024-01-11T11:08:18.092 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-11T11:08:18.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-11T11:08:18.213 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T11:08:18.207+0000 7f2a2446d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T11:08:18.486 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3053 16 374 358 0.007328 0 - 0.0803724 2024-01-11T11:08:19.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3054 16 374 358 0.0073256 0 - 0.0803724 2024-01-11T11:08:20.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3055 16 374 358 0.0073232 0 - 0.0803724 2024-01-11T11:08:21.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3056 16 374 358 0.00732081 0 - 0.0803724 2024-01-11T11:08:22.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3057 16 374 358 0.00731841 0 - 0.0803724 2024-01-11T11:08:23.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3058 16 374 358 0.00731602 0 - 0.0803724 2024-01-11T11:08:24.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3059 16 374 358 0.00731363 0 - 0.0803724 2024-01-11T11:08:25.487 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:08:25.486236+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:08:25.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:08:25.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3060 16 374 358 0.00731124 0 - 0.0803724 2024-01-11T11:08:26.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3061 16 374 358 0.00730885 0 - 0.0803724 2024-01-11T11:08:27.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3062 16 374 358 0.00730646 0 - 0.0803724 2024-01-11T11:08:28.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3063 16 374 358 0.00730408 0 - 0.0803724 2024-01-11T11:08:29.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3064 16 374 358 0.00730169 0 - 0.0803724 2024-01-11T11:08:30.488 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3065 16 374 358 0.00729931 0 - 0.0803724 2024-01-11T11:08:31.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3066 16 374 358 0.00729693 0 - 0.0803724 2024-01-11T11:08:32.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3067 16 374 358 0.00729455 0 - 0.0803724 2024-01-11T11:08:33.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3068 16 374 358 0.00729217 0 - 0.0803724 2024-01-11T11:08:34.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3069 16 374 358 0.0072898 0 - 0.0803724 2024-01-11T11:08:35.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3070 16 374 358 0.00728742 0 - 0.0803724 2024-01-11T11:08:36.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3071 16 374 358 0.00728505 0 - 0.0803724 2024-01-11T11:08:37.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3072 16 374 358 0.00728268 0 - 0.0803724 2024-01-11T11:08:38.489 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3073 16 374 358 0.00728031 0 - 0.0803724 2024-01-11T11:08:39.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3074 16 374 358 0.00727794 0 - 0.0803724 2024-01-11T11:08:40.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3075 16 374 358 0.00727557 0 - 0.0803724 2024-01-11T11:08:41.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3076 16 374 358 0.00727321 0 - 0.0803724 2024-01-11T11:08:42.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3077 16 374 358 0.00727084 0 - 0.0803724 2024-01-11T11:08:43.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3078 16 374 358 0.00726848 0 - 0.0803724 2024-01-11T11:08:44.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3079 16 374 358 0.00726612 0 - 0.0803724 2024-01-11T11:08:45.490 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:08:45.489132+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:08:45.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:08:45.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3080 16 374 358 0.00726376 0 - 0.0803724 2024-01-11T11:08:46.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3081 16 374 358 0.0072614 0 - 0.0803724 2024-01-11T11:08:47.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3082 16 374 358 0.00725905 0 - 0.0803724 2024-01-11T11:08:48.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3083 16 374 358 0.00725669 0 - 0.0803724 2024-01-11T11:08:49.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3084 16 374 358 0.00725434 0 - 0.0803724 2024-01-11T11:08:50.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3085 16 374 358 0.00725199 0 - 0.0803724 2024-01-11T11:08:51.491 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3086 16 374 358 0.00724964 0 - 0.0803724 2024-01-11T11:08:52.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3087 16 374 358 0.00724729 0 - 0.0803724 2024-01-11T11:08:53.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3088 16 374 358 0.00724494 0 - 0.0803724 2024-01-11T11:08:54.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3089 16 374 358 0.0072426 0 - 0.0803724 2024-01-11T11:08:55.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3090 16 374 358 0.00724025 0 - 0.0803724 2024-01-11T11:08:56.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3091 16 374 358 0.00723791 0 - 0.0803724 2024-01-11T11:08:57.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3092 16 374 358 0.00723557 0 - 0.0803724 2024-01-11T11:08:58.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3093 16 374 358 0.00723323 0 - 0.0803724 2024-01-11T11:08:59.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3094 16 374 358 0.00723089 0 - 0.0803724 2024-01-11T11:09:00.492 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3095 16 374 358 0.00722856 0 - 0.0803724 2024-01-11T11:09:01.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3096 16 374 358 0.00722622 0 - 0.0803724 2024-01-11T11:09:02.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3097 16 374 358 0.00722389 0 - 0.0803724 2024-01-11T11:09:03.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3098 16 374 358 0.00722156 0 - 0.0803724 2024-01-11T11:09:04.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3099 16 374 358 0.00721923 0 - 0.0803724 2024-01-11T11:09:05.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:09:05.491894+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:09:05.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:09:05.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3100 16 374 358 0.0072169 0 - 0.0803724 2024-01-11T11:09:06.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3101 16 374 358 0.00721457 0 - 0.0803724 2024-01-11T11:09:07.493 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3102 16 374 358 0.00721224 0 - 0.0803724 2024-01-11T11:09:08.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3103 16 374 358 0.00720992 0 - 0.0803724 2024-01-11T11:09:09.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3104 16 374 358 0.0072076 0 - 0.0803724 2024-01-11T11:09:10.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3105 16 374 358 0.00720528 0 - 0.0803724 2024-01-11T11:09:11.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3106 16 374 358 0.00720296 0 - 0.0803724 2024-01-11T11:09:12.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3107 16 374 358 0.00720064 0 - 0.0803724 2024-01-11T11:09:13.494 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3108 16 374 358 0.00719832 0 - 0.0803724 2024-01-11T11:09:14.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3109 16 374 358 0.00719601 0 - 0.0803724 2024-01-11T11:09:15.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3110 16 374 358 0.00719369 0 - 0.0803724 2024-01-11T11:09:16.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3111 16 374 358 0.00719138 0 - 0.0803724 2024-01-11T11:09:17.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3112 16 374 358 0.00718907 0 - 0.0803724 2024-01-11T11:09:18.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3113 16 374 358 0.00718676 0 - 0.0803724 2024-01-11T11:09:19.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3114 16 374 358 0.00718445 0 - 0.0803724 2024-01-11T11:09:20.495 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3115 16 374 358 0.00718214 0 - 0.0803724 2024-01-11T11:09:21.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3116 16 374 358 0.00717984 0 - 0.0803724 2024-01-11T11:09:22.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3117 16 374 358 0.00717754 0 - 0.0803724 2024-01-11T11:09:23.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3118 16 374 358 0.00717523 0 - 0.0803724 2024-01-11T11:09:24.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3119 16 374 358 0.00717293 0 - 0.0803724 2024-01-11T11:09:25.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:09:25.494719+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:09:25.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:09:25.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3120 16 374 358 0.00717063 0 - 0.0803724 2024-01-11T11:09:26.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3121 16 374 358 0.00716834 0 - 0.0803724 2024-01-11T11:09:27.496 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3122 16 374 358 0.00716604 0 - 0.0803724 2024-01-11T11:09:28.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3123 16 374 358 0.00716375 0 - 0.0803724 2024-01-11T11:09:29.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3124 16 374 358 0.00716145 0 - 0.0803724 2024-01-11T11:09:30.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3125 16 374 358 0.00715916 0 - 0.0803724 2024-01-11T11:09:31.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3126 16 374 358 0.00715687 0 - 0.0803724 2024-01-11T11:09:32.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3127 16 374 358 0.00715458 0 - 0.0803724 2024-01-11T11:09:33.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3128 16 374 358 0.00715229 0 - 0.0803724 2024-01-11T11:09:34.497 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3129 16 374 358 0.00715001 0 - 0.0803724 2024-01-11T11:09:35.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3130 16 374 358 0.00714772 0 - 0.0803724 2024-01-11T11:09:36.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3131 16 374 358 0.00714544 0 - 0.0803724 2024-01-11T11:09:37.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3132 16 374 358 0.00714316 0 - 0.0803724 2024-01-11T11:09:38.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3133 16 374 358 0.00714088 0 - 0.0803724 2024-01-11T11:09:39.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3134 16 374 358 0.0071386 0 - 0.0803724 2024-01-11T11:09:40.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3135 16 374 358 0.00713632 0 - 0.0803724 2024-01-11T11:09:41.498 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3136 16 374 358 0.00713405 0 - 0.0803724 2024-01-11T11:09:42.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3137 16 374 358 0.00713177 0 - 0.0803724 2024-01-11T11:09:43.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3138 16 374 358 0.0071295 0 - 0.0803724 2024-01-11T11:09:44.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3139 16 374 358 0.00712723 0 - 0.0803724 2024-01-11T11:09:45.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:09:45.497571+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:09:45.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:09:45.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3140 16 374 358 0.00712496 0 - 0.0803724 2024-01-11T11:09:46.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3141 16 374 358 0.00712269 0 - 0.0803724 2024-01-11T11:09:47.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3142 16 374 358 0.00712043 0 - 0.0803724 2024-01-11T11:09:48.499 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3143 16 374 358 0.00711816 0 - 0.0803724 2024-01-11T11:09:49.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3144 16 374 358 0.0071159 0 - 0.0803724 2024-01-11T11:09:50.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3145 16 374 358 0.00711363 0 - 0.0803724 2024-01-11T11:09:51.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3146 16 374 358 0.00711137 0 - 0.0803724 2024-01-11T11:09:52.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3147 16 374 358 0.00710911 0 - 0.0803724 2024-01-11T11:09:53.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3148 16 374 358 0.00710685 0 - 0.0803724 2024-01-11T11:09:54.500 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3149 16 374 358 0.0071046 0 - 0.0803724 2024-01-11T11:09:55.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3150 16 374 358 0.00710234 0 - 0.0803724 2024-01-11T11:09:56.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3151 16 374 358 0.00710009 0 - 0.0803724 2024-01-11T11:09:57.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3152 16 374 358 0.00709783 0 - 0.0803724 2024-01-11T11:09:58.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3153 16 374 358 0.00709558 0 - 0.0803724 2024-01-11T11:09:59.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3154 16 374 358 0.00709333 0 - 0.0803724 2024-01-11T11:10:00.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3155 16 374 358 0.00709109 0 - 0.0803724 2024-01-11T11:10:01.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3156 16 374 358 0.00708884 0 - 0.0803724 2024-01-11T11:10:02.501 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3157 16 374 358 0.00708659 0 - 0.0803724 2024-01-11T11:10:03.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3158 16 374 358 0.00708435 0 - 0.0803724 2024-01-11T11:10:04.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3159 16 374 358 0.00708211 0 - 0.0803724 2024-01-11T11:10:05.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:10:05.500276+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:10:05.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:10:05.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3160 16 374 358 0.00707987 0 - 0.0803724 2024-01-11T11:10:06.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3161 16 374 358 0.00707763 0 - 0.0803724 2024-01-11T11:10:07.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3162 16 374 358 0.00707539 0 - 0.0803724 2024-01-11T11:10:08.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3163 16 374 358 0.00707315 0 - 0.0803724 2024-01-11T11:10:09.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3164 16 374 358 0.00707091 0 - 0.0803724 2024-01-11T11:10:10.502 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3165 16 374 358 0.00706868 0 - 0.0803724 2024-01-11T11:10:11.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3166 16 374 358 0.00706645 0 - 0.0803724 2024-01-11T11:10:12.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3167 16 374 358 0.00706422 0 - 0.0803724 2024-01-11T11:10:13.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3168 16 374 358 0.00706199 0 - 0.0803724 2024-01-11T11:10:14.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3169 16 374 358 0.00705976 0 - 0.0803724 2024-01-11T11:10:15.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3170 16 374 358 0.00705753 0 - 0.0803724 2024-01-11T11:10:16.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3171 16 374 358 0.0070553 0 - 0.0803724 2024-01-11T11:10:17.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3172 16 374 358 0.00705308 0 - 0.0803724 2024-01-11T11:10:18.130 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:10:18.131 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-01-11T11:10:18.143 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-11T11:10:18.151 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-11T11:10:18.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-11T11:10:18.277 INFO:teuthology.orchestra.run.smithi040.stderr:2024-01-11T11:10:18.270+0000 7fd528b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-11T11:10:18.503 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3173 16 374 358 0.00705086 0 - 0.0803724 2024-01-11T11:10:19.504 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3174 16 374 358 0.00704864 0 - 0.0803724 2024-01-11T11:10:20.504 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3175 16 374 358 0.00704642 0 - 0.0803724 2024-01-11T11:10:21.504 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3176 16 374 358 0.0070442 0 - 0.0803724 2024-01-11T11:10:22.504 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3177 16 374 358 0.00704198 0 - 0.0803724 2024-01-11T11:10:23.504 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3178 16 374 358 0.00703976 0 - 0.0803724 2024-01-11T11:10:24.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3179 16 374 358 0.00703755 0 - 0.0803724 2024-01-11T11:10:25.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:10:25.503025+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:10:25.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:10:25.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3180 16 374 358 0.00703534 0 - 0.0803724 2024-01-11T11:10:26.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3181 16 374 358 0.00703313 0 - 0.0803724 2024-01-11T11:10:27.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3182 16 374 358 0.00703091 0 - 0.0803724 2024-01-11T11:10:28.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3183 16 374 358 0.00702871 0 - 0.0803724 2024-01-11T11:10:29.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3184 16 374 358 0.0070265 0 - 0.0803724 2024-01-11T11:10:30.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3185 16 374 358 0.00702429 0 - 0.0803724 2024-01-11T11:10:31.505 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3186 16 374 358 0.00702209 0 - 0.0803724 2024-01-11T11:10:32.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3187 16 374 358 0.00701988 0 - 0.0803724 2024-01-11T11:10:33.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3188 16 374 358 0.00701768 0 - 0.0803724 2024-01-11T11:10:34.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3189 16 374 358 0.00701548 0 - 0.0803724 2024-01-11T11:10:35.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3190 16 374 358 0.00701328 0 - 0.0803724 2024-01-11T11:10:36.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3191 16 374 358 0.00701108 0 - 0.0803724 2024-01-11T11:10:37.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3192 16 374 358 0.00700889 0 - 0.0803724 2024-01-11T11:10:38.506 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3193 16 374 358 0.00700669 0 - 0.0803724 2024-01-11T11:10:39.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3194 16 374 358 0.0070045 0 - 0.0803724 2024-01-11T11:10:40.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3195 16 374 358 0.00700231 0 - 0.0803724 2024-01-11T11:10:41.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3196 16 374 358 0.00700012 0 - 0.0803724 2024-01-11T11:10:42.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3197 16 374 358 0.00699793 0 - 0.0803724 2024-01-11T11:10:43.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3198 16 374 358 0.00699574 0 - 0.0803724 2024-01-11T11:10:44.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3199 16 374 358 0.00699355 0 - 0.0803724 2024-01-11T11:10:45.507 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:10:45.505765+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:10:45.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:10:45.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3200 16 374 358 0.00699136 0 - 0.0803724 2024-01-11T11:10:46.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3201 16 374 358 0.00698918 0 - 0.0803724 2024-01-11T11:10:47.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3202 16 374 358 0.006987 0 - 0.0803724 2024-01-11T11:10:48.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3203 16 374 358 0.00698482 0 - 0.0803724 2024-01-11T11:10:49.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3204 16 374 358 0.00698264 0 - 0.0803724 2024-01-11T11:10:50.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3205 16 374 358 0.00698046 0 - 0.0803724 2024-01-11T11:10:51.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3206 16 374 358 0.00697828 0 - 0.0803724 2024-01-11T11:10:52.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3207 16 374 358 0.0069761 0 - 0.0803724 2024-01-11T11:10:53.508 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3208 16 374 358 0.00697393 0 - 0.0803724 2024-01-11T11:10:54.509 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3209 16 374 358 0.00697176 0 - 0.0803724 2024-01-11T11:10:55.509 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3210 16 374 358 0.00696958 0 - 0.0803724 2024-01-11T11:10:56.509 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3211 16 374 358 0.00696741 0 - 0.0803724 2024-01-11T11:10:57.509 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3212 16 374 358 0.00696524 0 - 0.0803724 2024-01-11T11:10:58.509 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3213 16 374 358 0.00696308 0 - 0.0803724 2024-01-11T11:10:59.509 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3214 16 374 358 0.00696091 0 - 0.0803724 2024-01-11T11:11:00.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3215 16 374 358 0.00695875 0 - 0.0803724 2024-01-11T11:11:01.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3216 16 374 358 0.00695658 0 - 0.0803724 2024-01-11T11:11:02.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3217 16 374 358 0.00695442 0 - 0.0803724 2024-01-11T11:11:03.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3218 16 374 358 0.00695226 0 - 0.0803724 2024-01-11T11:11:04.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3219 16 374 358 0.0069501 0 - 0.0803724 2024-01-11T11:11:05.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:11:05.508605+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:11:05.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:11:05.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3220 16 374 358 0.00694794 0 - 0.0803724 2024-01-11T11:11:06.510 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3221 16 374 358 0.00694578 0 - 0.0803724 2024-01-11T11:11:07.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3222 16 374 358 0.00694363 0 - 0.0803724 2024-01-11T11:11:08.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3223 16 374 358 0.00694147 0 - 0.0803724 2024-01-11T11:11:09.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3224 16 374 358 0.00693932 0 - 0.0803724 2024-01-11T11:11:10.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3225 16 374 358 0.00693717 0 - 0.0803724 2024-01-11T11:11:11.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3226 16 374 358 0.00693502 0 - 0.0803724 2024-01-11T11:11:12.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3227 16 374 358 0.00693287 0 - 0.0803724 2024-01-11T11:11:13.511 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3228 16 374 358 0.00693072 0 - 0.0803724 2024-01-11T11:11:14.512 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3229 16 374 358 0.00692857 0 - 0.0803724 2024-01-11T11:11:15.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3230 16 374 358 0.00692643 0 - 0.0803724 2024-01-11T11:11:16.512 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3231 16 374 358 0.00692428 0 - 0.0803724 2024-01-11T11:11:17.512 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3232 16 374 358 0.00692214 0 - 0.0803724 2024-01-11T11:11:18.512 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3233 16 374 358 0.00692 0 - 0.0803724 2024-01-11T11:11:19.512 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3234 16 374 358 0.00691786 0 - 0.0803724 2024-01-11T11:11:20.513 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3235 16 374 358 0.00691572 0 - 0.0803724 2024-01-11T11:11:21.513 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3236 16 374 358 0.00691359 0 - 0.0803724 2024-01-11T11:11:22.513 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3237 16 374 358 0.00691145 0 - 0.0803724 2024-01-11T11:11:23.513 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3238 16 374 358 0.00690931 0 - 0.0803724 2024-01-11T11:11:24.513 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3239 16 374 358 0.00690718 0 - 0.0803724 2024-01-11T11:11:25.513 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:11:25.511564+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:11:25.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:11:25.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3240 16 374 358 0.00690505 0 - 0.0803724 2024-01-11T11:11:26.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3241 16 374 358 0.00690292 0 - 0.0803724 2024-01-11T11:11:27.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3242 16 374 358 0.00690079 0 - 0.0803724 2024-01-11T11:11:28.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3243 16 374 358 0.00689866 0 - 0.0803724 2024-01-11T11:11:29.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3244 16 374 358 0.00689654 0 - 0.0803724 2024-01-11T11:11:30.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3245 16 374 358 0.00689441 0 - 0.0803724 2024-01-11T11:11:31.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3246 16 374 358 0.00689229 0 - 0.0803724 2024-01-11T11:11:32.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3247 16 374 358 0.00689016 0 - 0.0803724 2024-01-11T11:11:33.514 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3248 16 374 358 0.00688804 0 - 0.0803724 2024-01-11T11:11:34.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3249 16 374 358 0.00688592 0 - 0.0803724 2024-01-11T11:11:35.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3250 16 374 358 0.0068838 0 - 0.0803724 2024-01-11T11:11:36.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3251 16 374 358 0.00688169 0 - 0.0803724 2024-01-11T11:11:37.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3252 16 374 358 0.00687957 0 - 0.0803724 2024-01-11T11:11:38.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3253 16 374 358 0.00687745 0 - 0.0803724 2024-01-11T11:11:39.515 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3254 16 374 358 0.00687534 0 - 0.0803724 2024-01-11T11:11:40.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3255 16 374 358 0.00687323 0 - 0.0803724 2024-01-11T11:11:41.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3256 16 374 358 0.00687112 0 - 0.0803724 2024-01-11T11:11:42.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3257 16 374 358 0.00686901 0 - 0.0803724 2024-01-11T11:11:43.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3258 16 374 358 0.0068669 0 - 0.0803724 2024-01-11T11:11:44.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3259 16 374 358 0.00686479 0 - 0.0803724 2024-01-11T11:11:45.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:11:45.514482+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:11:45.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:11:45.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3260 16 374 358 0.00686269 0 - 0.0803724 2024-01-11T11:11:46.516 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3261 16 374 358 0.00686058 0 - 0.0803724 2024-01-11T11:11:47.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3262 16 374 358 0.00685848 0 - 0.0803724 2024-01-11T11:11:48.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3263 16 374 358 0.00685638 0 - 0.0803724 2024-01-11T11:11:49.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3264 16 374 358 0.00685428 0 - 0.0803724 2024-01-11T11:11:50.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3265 16 374 358 0.00685218 0 - 0.0803724 2024-01-11T11:11:51.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3266 16 374 358 0.00685008 0 - 0.0803724 2024-01-11T11:11:52.517 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3267 16 374 358 0.00684798 0 - 0.0803724 2024-01-11T11:11:53.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3268 16 374 358 0.00684589 0 - 0.0803724 2024-01-11T11:11:54.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3269 16 374 358 0.00684379 0 - 0.0803724 2024-01-11T11:11:55.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3270 16 374 358 0.0068417 0 - 0.0803724 2024-01-11T11:11:56.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3271 16 374 358 0.00683961 0 - 0.0803724 2024-01-11T11:11:57.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3272 16 374 358 0.00683752 0 - 0.0803724 2024-01-11T11:11:58.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3273 16 374 358 0.00683543 0 - 0.0803724 2024-01-11T11:11:59.518 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3274 16 374 358 0.00683334 0 - 0.0803724 2024-01-11T11:12:00.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3275 16 374 358 0.00683125 0 - 0.0803724 2024-01-11T11:12:01.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3276 16 374 358 0.00682917 0 - 0.0803724 2024-01-11T11:12:02.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3277 16 374 358 0.00682708 0 - 0.0803724 2024-01-11T11:12:03.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3278 16 374 358 0.006825 0 - 0.0803724 2024-01-11T11:12:04.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3279 16 374 358 0.00682292 0 - 0.0803724 2024-01-11T11:12:05.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:12:05.517423+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:12:05.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:12:05.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3280 16 374 358 0.00682084 0 - 0.0803724 2024-01-11T11:12:06.519 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3281 16 374 358 0.00681876 0 - 0.0803724 2024-01-11T11:12:07.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3282 16 374 358 0.00681668 0 - 0.0803724 2024-01-11T11:12:08.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3283 16 374 358 0.00681461 0 - 0.0803724 2024-01-11T11:12:09.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3284 16 374 358 0.00681253 0 - 0.0803724 2024-01-11T11:12:10.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3285 16 374 358 0.00681046 0 - 0.0803724 2024-01-11T11:12:11.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3286 16 374 358 0.00680838 0 - 0.0803724 2024-01-11T11:12:12.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3287 16 374 358 0.00680631 0 - 0.0803724 2024-01-11T11:12:13.520 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3288 16 374 358 0.00680424 0 - 0.0803724 2024-01-11T11:12:14.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3289 16 374 358 0.00680217 0 - 0.0803724 2024-01-11T11:12:15.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3290 16 374 358 0.00680011 0 - 0.0803724 2024-01-11T11:12:16.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3291 16 374 358 0.00679804 0 - 0.0803724 2024-01-11T11:12:17.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3292 16 374 358 0.00679598 0 - 0.0803724 2024-01-11T11:12:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-11T11:12:18.194 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-01-11T11:12:18.214 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-11T11:12:18.214 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-11T11:12:18.215 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-11T11:12:18.215 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T11:12:18.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3293 16 374 358 0.00679391 0 - 0.0803724 2024-01-11T11:12:19.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3294 16 374 358 0.00679185 0 - 0.0803724 2024-01-11T11:12:20.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3295 16 374 358 0.00678979 0 - 0.0803724 2024-01-11T11:12:21.521 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3296 16 374 358 0.00678773 0 - 0.0803724 2024-01-11T11:12:22.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3297 16 374 358 0.00678567 0 - 0.0803724 2024-01-11T11:12:23.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3298 16 374 358 0.00678361 0 - 0.0803724 2024-01-11T11:12:24.317 INFO:tasks.ceph.osd.0:Stopped 2024-01-11T11:12:24.317 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-11T11:12:24.317 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T11:12:24.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3299 16 374 358 0.00678156 0 - 0.0803724 2024-01-11T11:12:25.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:12:25.520191+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:12:25.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:12:25.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3300 16 374 358 0.0067795 0 - 0.0803724 2024-01-11T11:12:26.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3301 16 374 358 0.00677745 0 - 0.0803724 2024-01-11T11:12:27.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3302 16 374 358 0.00677539 0 - 0.0803724 2024-01-11T11:12:28.522 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3303 16 374 358 0.00677334 0 - 0.0803724 2024-01-11T11:12:29.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3304 16 374 358 0.00677129 0 - 0.0803724 2024-01-11T11:12:30.418 INFO:tasks.ceph.osd.1:Stopped 2024-01-11T11:12:30.419 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-11T11:12:30.419 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T11:12:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T11:12:30.419 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2024-01-11T11:12:30.437 INFO:tasks.ceph.osd.2:Stopped 2024-01-11T11:12:30.438 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-11T11:12:30.438 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T11:12:30.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3305 16 374 358 0.00676924 0 - 0.0803724 2024-01-11T11:12:30.532 INFO:tasks.ceph.osd.3:Stopped 2024-01-11T11:12:30.532 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-11T11:12:30.533 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-11T11:12:30.533 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T11:12:30.533 INFO:tasks.ceph.mgr.x:Stopped 2024-01-11T11:12:30.533 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-11T11:12:30.533 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-11T11:12:30.533 INFO:teuthology.orchestra.run:waiting for 300 2024-01-11T11:12:30.533 INFO:tasks.ceph.mon.a:Stopped 2024-01-11T11:12:30.534 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-11T11:12:30.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2024-01-11T11:12:30.571 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T11:12:30.571 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-11T11:12:30.729 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T11:12:30.729 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-11T11:12:30.807 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T11:12:30.807 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-11T11:12:30.882 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T11:12:30.882 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-11T11:12:30.946 INFO:tasks.ceph:Archiving mon data... 2024-01-11T11:12:30.948 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/data/mon.a.tgz 2024-01-11T11:12:30.948 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-01-11T11:12:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.5UiK4X6GwM 2024-01-11T11:12:30.966 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.5UiK4X6GwM 2024-01-11T11:12:31.087 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.5UiK4X6GwM 2024-01-11T11:12:31.173 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.5UiK4X6GwM is 236KB 2024-01-11T11:12:31.266 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.5UiK4X6GwM 2024-01-11T11:12:31.280 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-11T11:12:31.280 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-11T11:12:31.322 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-11T11:12:31.400 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-11T11:12:31.417 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-11T11:12:31.417 INFO:tasks.ceph:Archiving crash dumps... 2024-01-11T11:12:31.418 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/remote/smithi040/crash 2024-01-11T11:12:31.419 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-11T11:12:31.466 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/remote/smithi134/crash 2024-01-11T11:12:31.467 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-11T11:12:31.504 INFO:tasks.ceph:Compressing logs... 2024-01-11T11:12:31.504 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-11T11:12:31.511 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-11T11:12:31.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3306 16 374 358 0.0067672 0 - 0.0803724 2024-01-11T11:12:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42335.log 2024-01-11T11:12:31.534 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52762.log 2024-01-11T11:12:31.585 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28100.log 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28220.log 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28180.log/var/log/ceph/ceph-client.admin.28100.log: 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28100.log.gz 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.28220.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28220.log.gz 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.28180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28140.log 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28180.log.gz 2024-01-11T11:12:31.586 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-11T11:12:31.587 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.28140.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-11T11:12:31.587 INFO:teuthology.orchestra.run.smithi134.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28140.log.gz 2024-01-11T11:12:31.642 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-osd.2.log: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-11T11:12:31.818 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42335.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27417.log 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42335.log.gz 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47578.log 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52762.log.gz 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.tmp-client.admin.27417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37575.log 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr: 66.9% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27417.log.gz 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50527.log 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47578.log.gz 2024-01-11T11:12:31.819 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41458.log 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37575.log.gz 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48359.log 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50527.log.gz 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53230.log 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41458.log.gz 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52929.log 2024-01-11T11:12:31.820 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48359.log.gz 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51532.log 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53230.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53230.log.gz 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.52929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28473.log 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51532.log.gz 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40890.log 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36087.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37357.log 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28473.log.gz 2024-01-11T11:12:31.821 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33937.log 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40890.log.gz 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38729.log 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37357.log.gz 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33937.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33937.log.gz 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49733.log 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38729.log.gz 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38918.log 2024-01-11T11:12:31.822 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51378.log 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49733.log.gz 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38918.log: gzip -5 --verbose -- 50.4% /var/log/ceph/ceph-client.admin.46689.log 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38918.log.gz 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51815.log 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51378.log.gz 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46689.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.46689.log.gz 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32945.log 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47440.log 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51815.log.gz 2024-01-11T11:12:31.823 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32945.log: gzip -5 50.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.32945.log.gz 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.42780.log 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52720.log 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.47440.log.gz 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42780.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42780.log.gz 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45396.log 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50170.log 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52720.log.gz 2024-01-11T11:12:31.824 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45396.log: gzip -5 --verbose 50.8% -- -- replaced with /var/log/ceph/ceph-client.admin.45396.log.gz /var/log/ceph/ceph-client.admin.45673.log 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45532.log 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50170.log.gz 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45673.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45673.log.gz 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30653.log 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29016.log 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45532.log.gz 2024-01-11T11:12:31.825 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30653.log: gzip 49.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.30653.log.gz 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53320.log 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33510.log 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.29016.log.gz 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47304.log 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53320.log.gz 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46943.log 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33510.log.gz 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47304.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47304.log.gz 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39882.log 2024-01-11T11:12:31.826 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45316.log 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46943.log.gz 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39882.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39882.log.gz 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41777.log 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33728.log 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45316.log.gz 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41777.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41777.log.gz 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47712.log 2024-01-11T11:12:31.827 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37437.log 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33728.log.gz 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47712.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47712.log.gz 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46558.log 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52312.log 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37437.log.gz 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46558.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46558.log.gz 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50386.log 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37486.log 2024-01-11T11:12:31.828 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.52312.log.gz 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50386.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50386.log.gz 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46903.log 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36614.log 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37486.log.gz 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46903.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46903.log.gz 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30284.log 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52432.log 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36614.log.gz 2024-01-11T11:12:31.829 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30284.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30284.log.gz 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30410.log 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41819.log 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52432.log.gz 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30410.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30410.log.gz 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32500.log 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47487.log 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41819.log.gz 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32500.log: gzip -5 50.4% --verbose -- replaced with /var/log/ceph/ceph-client.admin.32500.log.gz -- 2024-01-11T11:12:31.830 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.34266.log 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30941.log 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47487.log.gz 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34266.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34266.log.gz 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53612.log 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32856.log 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.30941.log: /var/log/ceph/ceph-client.admin.53612.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53612.log.gz 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43144.log 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38380.log 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32856.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32856.log.gz 2024-01-11T11:12:31.831 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43144.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43144.log.gz 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34035.log 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38051.log 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38380.log.gz 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31674.log 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34035.log.gz 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50250.log 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38051.log.gz 2024-01-11T11:12:31.832 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40843.log 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31674.log.gz 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43300.log 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50250.log.gz 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51338.log 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40843.log.gz 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29738.log 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43300.log.gz 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34624.log 2024-01-11T11:12:31.833 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.51338.log.gz 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48613.log 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29738.log.gz 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48134.log 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34624.log.gz 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39236.log 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48613.log.gz 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51679.log 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48134.log.gz 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30452.log 2024-01-11T11:12:31.834 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39236.log.gz 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48486.log 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51679.log.gz 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49307.log 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30452.log.gz 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44144.log 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48486.log.gz 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31305.log 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49307.log.gz 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51773.log 2024-01-11T11:12:31.835 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44144.log.gz 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31127.log 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31305.log.gz 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51773.log.gz 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52176.log 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31127.log.gz 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37962.log 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2024-01-11T11:12:31.836 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48831.log 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52176.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52176.log.gz 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37962.log: gzip 50.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.29630.log 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37962.log.gz 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48007.log 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48831.log.gz 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36545.log 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29630.log.gz 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28708.log 2024-01-11T11:12:31.837 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48007.log.gz 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35442.log 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36545.log.gz 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28708.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28708.log.gz 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38500.log 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32420.log 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35442.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35442.log.gz 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38500.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38500.log.gz 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44626.log 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51425.log 2024-01-11T11:12:31.838 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32420.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32420.log.gz 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52134.log 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44626.log.gz 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50297.log 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51425.log.gz 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36307.log 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52134.log.gz 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50297.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50297.log.gz 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35820.log 2024-01-11T11:12:31.839 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44371.log 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36307.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36307.log.gz 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43095.log 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35820.log.gz 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39616.log 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44371.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44371.log.gz 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46380.log 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43095.log.gz 2024-01-11T11:12:31.840 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39616.log.gz 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39136.log 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46380.log.gz 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51512.log 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33332.log 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39136.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39136.log.gz 2024-01-11T11:12:31.841 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44060.log 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51512.log.gz 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30368.log 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33332.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33332.log.gz 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31445.log 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44060.log.gz 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30368.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30368.log.gz 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43581.log 2024-01-11T11:12:31.842 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48243.log 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31445.log.gz 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45893.log 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43581.log.gz 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30074.log 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48243.log.gz 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48312.log 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45893.log.gz 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30074.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30074.log.gz 2024-01-11T11:12:31.843 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43487.log 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46640.log 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48312.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48312.log.gz 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43487.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43487.log.gz 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49781.log 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46640.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46640.log.gz 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39343.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2024-01-11T11:12:31.844 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29906.log 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.49781.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37793.log 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49781.log.gz 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29906.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29906.log.gz 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46207.log 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40538.log 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37793.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37793.log.gz 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46207.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46207.log.gz 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50090.log 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52352.log 2024-01-11T11:12:31.845 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40538.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40538.log.gz 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33590.log 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50090.log.gz 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35651.log 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52352.log.gz 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33590.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41688.log 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33590.log.gz 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35651.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35651.log.gz 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49387.log 2024-01-11T11:12:31.846 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45055.log 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41688.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41688.log.gz 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49387.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49387.log.gz 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40240.log 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50043.log 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45055.log: 94.7% -- replaced with /var/log/ceph/ceph-client.0.52929.log.gz 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45055.log.gz 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43407.log 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40240.log.gz 2024-01-11T11:12:31.847 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46511.log 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr: 51.2% -- replaced with /var/log/ceph/ceph-client.admin.50043.log.gz 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28552.log 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43407.log.gz 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46511.log.gz 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52218.log 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28552.log.gz 2024-01-11T11:12:31.848 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34406.log 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46464.log 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52218.log.gz 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38331.log 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34406.log.gz 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36445.log 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46464.log.gz 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49133.log 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38331.log.gz 2024-01-11T11:12:31.849 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36356.log 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36445.log.gz 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47398.log 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49133.log.gz 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39392.log 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36356.log.gz 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32460.log 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47398.log.gz 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33759.log 2024-01-11T11:12:31.850 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.39392.log.gz 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31087.log 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32460.log.gz 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50130.log 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33759.log.gz 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35780.log 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31087.log.gz 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38191.log 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50130.log.gz 2024-01-11T11:12:31.851 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31536.log 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35780.log.gz 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46076.log 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38191.log.gz 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38778.log 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31536.log.gz 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29150.log 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46076.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46076.log.gz 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44547.log 2024-01-11T11:12:31.852 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38778.log.gz 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35860.log 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29150.log.gz 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.44547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37833.log 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35860.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35860.log.gz 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37010.log 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34855.log 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37833.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37833.log.gz 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37010.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37010.log.gz 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53139.log 2024-01-11T11:12:31.853 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43534.log 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34855.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34855.log.gz 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53841.log 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.53139.log.gz 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42201.log 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45189.log 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.53841.log.gz 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38240.log 2024-01-11T11:12:31.854 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42201.log.gz 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35273.log 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45189.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45189.log.gz 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53676.log 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38240.log.gz 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35602.log 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35273.log.gz 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33210.log 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53676.log.gz 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52087.log 2024-01-11T11:12:31.855 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35602.log.gz 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40071.log 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33210.log.gz 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41155.log 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52087.log.gz 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.39286.log 49.6% 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40071.log.gz 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2024-01-11T11:12:31.856 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41155.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41155.log.gz 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.0.39286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37228.log 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46783.log 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37228.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37228.log.gz 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.33815.log 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46783.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46783.log.gz 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37615.log 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48087.log 2024-01-11T11:12:31.857 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33815.log.gz 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44935.log 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37615.log.gz 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33065.log 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48087.log.gz 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44935.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.52902.log -- replaced with /var/log/ceph/ceph-client.admin.44935.log.gz 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33065.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33065.log.gz 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29967.log 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52902.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52902.log.gz 2024-01-11T11:12:31.858 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33630.log 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29967.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29967.log.gz 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35562.log 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33630.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33630.log.gz 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35562.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35562.log.gz 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50480.log 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40105.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36832.log 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50480.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50480.log.gz 2024-01-11T11:12:31.859 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36832.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36832.log.gz 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39524.log 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41505.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53412.log 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39524.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39524.log.gz 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49581.log 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53412.log: gzip -5 --verbose -- 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53412.log.gz 2024-01-11T11:12:31.860 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.51472.log 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49581.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49581.log.gz 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32062.log 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51472.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51472.log.gz 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43966.log 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32062.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32062.log.gz 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43966.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43966.log.gz 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38869.log 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41641.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-01-11T11:12:31.861 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34582.log 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38869.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38869.log.gz 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53109.log 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34582.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34582.log.gz 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30494.log 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53109.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53109.log.gz 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30494.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30494.log.gz 2024-01-11T11:12:31.862 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47126.log 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47351.log 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47126.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47126.log.gz 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51258.log 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47351.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47351.log.gz 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45236.log 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51258.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51258.log.gz 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2024-01-11T11:12:31.863 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45236.log.gz 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29421.log 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45142.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39749.log 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29421.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29421.log.gz 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53260.log 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39749.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39749.log.gz 2024-01-11T11:12:31.864 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42288.log 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53260.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53260.log.gz 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42288.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42288.log.gz 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51218.log 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42513.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.42161.log 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51218.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51218.log.gz 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35740.log 2024-01-11T11:12:31.865 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42161.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42161.log.gz 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29294.log 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35740.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35740.log.gz 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37695.log 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29294.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29294.log.gz 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37695.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37695.log.gz 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46034.log 2024-01-11T11:12:31.866 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49865.log 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46034.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46034.log.gz 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47927.log 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44498.log 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49865.log.gz 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47927.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47927.log.gz 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.42869.log 2024-01-11T11:12:31.867 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44498.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44498.log.gz 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46138.log 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46118.log 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42869.log.gz 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46138.log: gzip -5 --verbose -- 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46138.log.gz 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.43233.log 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46118.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46118.log.gz 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37853.log 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36663.log 2024-01-11T11:12:31.868 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43233.log.gz 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50210.log 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37853.log.gz 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42054.log 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36663.log.gz 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50210.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50210.log.gz 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.39574.log 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42054.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42054.log.gz 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34539.log 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39574.log: gzip 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39574.log.gz 2024-01-11T11:12:31.869 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50935.log 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34539.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34539.log.gz 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29798.log 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50935.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50935.log.gz 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53798.log 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29798.log.gz 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53798.log: gzip -5 --verbose -- 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53798.log.gz 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.44546.log 2024-01-11T11:12:31.870 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41992.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36127.log 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44546.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44546.log.gz 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.42963.log 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36127.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36127.log.gz 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30784.log 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42963.log: gzip -5 --verbose 50.4% -- -- replaced with /var/log/ceph/ceph-client.admin.42963.log.gz /var/log/ceph/ceph-client.admin.52472.log 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30784.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30784.log.gz 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46823.log 2024-01-11T11:12:31.871 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52472.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52472.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46736.log 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37179.log 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46823.log.gz 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37744.log 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46736.log.gz 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37179.log: gzip -5 --verbose -- 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37179.log.gz 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.29462.log 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37744.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37744.log.gz 2024-01-11T11:12:31.872 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29462.log: gzip -5 --verbose 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29462.log.gz 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.32767.log 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40498.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49004.log 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32767.log.gz 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50344.log 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49004.log.gz 2024-01-11T11:12:31.873 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41908.log 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47795.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50344.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50344.log.gz 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41908.log.gz 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45276.log 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51593.log 2024-01-11T11:12:31.874 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42822.log: /var/log/ceph/ceph-client.admin.45276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51298.log 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45276.log.gz 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29968.log 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51593.log.gz 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53903.log 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51298.log.gz 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53933.log 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29968.log: /var/log/ceph/ceph-client.admin.53903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41216.log 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53903.log.gz 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29968.log.gz 2024-01-11T11:12:31.875 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32985.log 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53933.log.gz 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53170.log 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41216.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41216.log.gz 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.32985.log: -- /var/log/ceph/ceph-client.admin.32598.log 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32985.log.gz 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53170.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53170.log.gz 2024-01-11T11:12:31.876 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45757.log 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32598.log: gzip -5 --verbose -- 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32598.log.gz 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.50616.log 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45757.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.45757.log.gz 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39657.log 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50616.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50616.log.gz 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36405.log 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39657.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39657.log.gz 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38002.log 2024-01-11T11:12:31.877 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36405.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31754.log 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36405.log.gz 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38002.log.gz 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31754.log: gzip -5 --verbose -- 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31754.log.gz 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-osd.1.log 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45804.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51726.log 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34077.log 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51726.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51726.log.gz 2024-01-11T11:12:31.878 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33292.log 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34077.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34077.log.gz 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32340.log 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33292.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33292.log.gz 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37248.log 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32340.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32340.log.gz 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47816.log 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49927.log 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37248.log: gzip/var/log/ceph/ceph-client.admin.47816.log: -5 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37248.log.gz 2024-01-11T11:12:31.879 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50743.log 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47816.log.gz 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49927.log: -- /var/log/ceph/ceph-client.admin.53442.log 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49927.log.gz/var/log/ceph/ceph-client.admin.50743.log: 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50743.log.gz 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45715.log 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53442.log: gzip -5 --verbose -- 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53442.log.gz 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.50569.log 2024-01-11T11:12:31.880 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45715.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45715.log.gz 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40160.log 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31794.log 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50569.log: /var/log/ceph/ceph-client.admin.40160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44698.log 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50569.log.gz 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31794.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31794.log.gz 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29461.log 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44191.log 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44698.log: /var/log/ceph/ceph-client.admin.29461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53643.log 2024-01-11T11:12:31.881 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44698.log.gz 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29461.log.gz 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44191.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44191.log.gz 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36167.log 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38282.log 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36167.log: /var/log/ceph/ceph-client.admin.53643.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.36167.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.44281.log 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53643.log.gz 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38282.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38282.log.gz 2024-01-11T11:12:31.882 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41068.log 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44281.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44281.log.gz 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.30827.log 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41068.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41068.log.gz 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30827.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51084.log 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30827.log.gz 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53352.log 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2024-01-11T11:12:31.883 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42377.log 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51084.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51084.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36265.log 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53352.log: /var/log/ceph/ceph-client.admin.42377.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53352.log.gz 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42377.log.gz 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36265.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.36265.log.gz 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35700.log 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40280.log: -- /var/log/ceph/ceph-client.admin.42466.log 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz/var/log/ceph/ceph-client.admin.35700.log: 2024-01-11T11:12:31.884 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35700.log.gz 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44324.log 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42466.log: -- /var/log/ceph/ceph-client.admin.42556.log 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42466.log.gz/var/log/ceph/ceph-client.admin.44324.log: 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44324.log.gz 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49347.log 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42556.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42556.log.gz 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.33550.log 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49347.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49347.log.gz 2024-01-11T11:12:31.885 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30326.log 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33550.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33897.log 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33550.log.gz 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46249.log 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30326.log.gz 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29631.log 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33897.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33897.log.gz 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47967.log 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46249.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46249.log.gz 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29631.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29631.log.gz 2024-01-11T11:12:31.886 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44238.log 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47967.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47967.log.gz 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.32727.log 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44238.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44238.log.gz 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34126.log 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32727.log: gzip -5 --verbose 50.4% -- /var/log/ceph/ceph-client.admin.40200.log 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32727.log.gz 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29192.log 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34126.log.gz 2024-01-11T11:12:31.887 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.40200.log: --verbose -- /var/log/ceph/ceph-client.admin.52820.log 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40200.log.gz/var/log/ceph/ceph-client.admin.29192.log: 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29192.log.gz 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40636.log 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52820.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.52820.log.gz 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48240.log 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40636.log: -- /var/log/ceph/ceph-client.admin.46990.log 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40636.log.gz/var/log/ceph/ceph-client.admin.48240.log: 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48240.log.gz 2024-01-11T11:12:31.888 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41369.log 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46990.log: gzip -5 --verbose 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46990.log.gz 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.31256.log 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41369.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43005.log 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41369.log.gz 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41552.log 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31256.log.gz 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49173.log 2024-01-11T11:12:31.889 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43005.log.gz 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47079.log 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41552.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41552.log.gz 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49173.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49173.log.gz 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43186.log 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47079.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47079.log.gz 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.30114.log 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43186.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43186.log.gz 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45940.log 2024-01-11T11:12:31.890 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30114.log: 49.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29291.log 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30114.log.gz 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36712.log 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45940.log.gz 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43047.log 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29291.log: /var/log/ceph/ceph-client.admin.36712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39973.log 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36712.log.gz 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29291.log.gz 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43047.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43047.log.gz 2024-01-11T11:12:31.891 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31634.log 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39003.log 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29570.log 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31634.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39973.log.gz 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31634.log.gz 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39003.log: gzip 50.8% -- replaced with /var/log/ceph/ceph-client.admin.39003.log.gz 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38142.log 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29570.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29570.log.gz 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39833.log 2024-01-11T11:12:31.892 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37050.log 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38142.log: /var/log/ceph/ceph-client.admin.39833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35035.log 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38142.log.gz 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39833.log.gz 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37050.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37050.log.gz 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46422.log 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35035.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35035.log.gz 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53767.log 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46422.log: -- /var/log/ceph/ceph-client.admin.38960.log 2024-01-11T11:12:31.893 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46422.log.gz/var/log/ceph/ceph-client.admin.53767.log: 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53767.log.gz 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45015.log 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38960.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38960.log.gz 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48791.log 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40022.log 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48791.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45015.log.gz 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48791.log.gz 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.894 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39216.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34315.log 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40022.log: gzip -5 --verbose 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40022.log.gz 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.47534.log 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34315.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34315.log.gz 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37916.log 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47534.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47534.log.gz 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.35522.log 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37916.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.37916.log.gz 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-01-11T11:12:31.895 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32687.log 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35522.log: /var/log/ceph/ceph-client.admin.41195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33252.log 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35522.log.gz 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32687.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32687.log.gz 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40418.log 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33252.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33252.log.gz 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36047.log 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40418.log: -- /var/log/ceph/ceph-client.admin.35253.log 2024-01-11T11:12:31.896 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4%/var/log/ceph/ceph-client.admin.36047.log: -- replaced with /var/log/ceph/ceph-client.admin.40418.log.gz 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47623.log 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36047.log.gz 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.35253.log: --verbose -- /var/log/ceph/ceph-client.admin.48446.log 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35253.log.gz 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47623.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47623.log.gz 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37397.log 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48446.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48446.log.gz 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34217.log 2024-01-11T11:12:31.897 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53736.log 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37397.log: /var/log/ceph/ceph-client.admin.34217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34355.log 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34217.log.gz 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37397.log.gz 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53736.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53736.log.gz 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43621.log 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34355.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34355.log.gz 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49907.log 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43621.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43621.log.gz 2024-01-11T11:12:31.898 INFO:teuthology.orchestra.run.smithi040.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31976.log 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49907.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49907.log.gz 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51909.log 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53017.log 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31976.log: gzip/var/log/ceph/ceph-client.admin.51909.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31207.log 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31976.log.gz 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51909.log.gz 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53017.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53017.log.gz 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51635.log 2024-01-11T11:12:31.899 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48181.log 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31207.log.gz 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53507.log 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51635.log.gz 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43447.log 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48181.log: /var/log/ceph/ceph-client.admin.53507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34706.log 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53507.log.gz 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48181.log.gz 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43447.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43447.log.gz 2024-01-11T11:12:31.900 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52987.log 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53706.log 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34706.log.gz 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52987.log: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gz 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38460.log 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49213.log 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53706.log: /var/log/ceph/ceph-client.admin.38460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32151.log 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr: 8.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38460.log.gz 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53706.log.gz 2024-01-11T11:12:31.901 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49213.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49213.log.gz 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43661.log 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32151.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32151.log.gz 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42602.log 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53873.log 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43661.log: /var/log/ceph/ceph-client.admin.42602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33372.log 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43661.log.gz 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42602.log.gz 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53873.log: 9.6% -- replaced with /var/log/ceph/ceph-client.admin.53873.log.gz 2024-01-11T11:12:31.902 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32647.log 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52699.log 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33372.log: /var/log/ceph/ceph-client.admin.32647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45851.log 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33372.log.gz 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32647.log.gz 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52699.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52699.log.gz 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33679.log 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45851.log: /var/log/ceph/ceph-client.admin.44013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35909.log 2024-01-11T11:12:31.903 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45851.log.gz 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33679.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33679.log.gz 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52566.log 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35909.log: /var/log/ceph/ceph-client.admin.52566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40378.log 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35909.log.gz 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52566.log.gz 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-01-11T11:12:31.904 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.45356.log 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40378.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40378.log.gz 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36792.log 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45356.log: gzip -5 --verbose -- 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45356.log.gz 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr: /var/log/ceph/ceph-client.admin.47885.log 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36792.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36792.log.gz 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39441.log 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52265.log 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47885.log.gz 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31843.log 2024-01-11T11:12:31.905 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39441.log.gz 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50998.log 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52265.log: /var/log/ceph/ceph-client.admin.31843.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52265.log.gz 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35075.log 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31843.log.gz 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48964.log 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50998.log: /var/log/ceph/ceph-client.admin.35075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44888.log 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50998.log.gz 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35075.log.gz/var/log/ceph/ceph-client.admin.48964.log: 2024-01-11T11:12:31.906 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51951.log 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48964.log.gz 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28736.log 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44888.log: /var/log/ceph/ceph-client.admin.51951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35204.log 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51951.log.gz 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44888.log.gz 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28736.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28736.log.gz 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45077.log 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35204.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.35204.log.gz 2024-01-11T11:12:31.907 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29292.log 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.45077.log: /var/log/ceph/ceph-client.admin.39054.log 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29292.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45077.log.gz 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42241.log 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29292.log.gz 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.39054.log: --verbose -- /var/log/ceph/ceph-client.admin.48047.log 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39054.log.gz/var/log/ceph/ceph-client.admin.42241.log: 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42241.log.gz 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43741.log 2024-01-11T11:12:31.908 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48047.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48047.log.gz 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52519.log 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5/var/log/ceph/ceph-client.admin.43741.log: --verbose -- /var/log/ceph/ceph-client.admin.40677.log 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0%/var/log/ceph/ceph-client.admin.52519.log: -- replaced with /var/log/ceph/ceph-client.admin.43741.log.gz 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44975.log 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52519.log.gz 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip/var/log/ceph/ceph-client.admin.40677.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32549.log 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.40677.log.gz 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44975.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44975.log.gz 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46600.log 2024-01-11T11:12:31.909 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.32549.log: -- /var/log/ceph/ceph-client.admin.30741.log 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32549.log.gz/var/log/ceph/ceph-client.admin.46600.log: 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46600.log.gz 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40587.log 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30741.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30741.log.gz 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49048.log 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35115.log 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38540.log 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49048.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 50.0% 2024-01-11T11:12:31.910 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49048.log.gz 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35115.log: gzip 50.8% -5 --verbose -- /var/log/ceph/ceph-client.admin.31345.log 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35115.log.gz 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38540.log: gzip -5 --verbose 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38540.log.gz 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr: -- /var/log/ceph/ceph-client.admin.37130.log 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31345.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31345.log.gz 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29926.log 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50703.log 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37130.log: gzip/var/log/ceph/ceph-client.admin.29926.log: -5 50.4% -- replaced with /var/log/ceph/ceph-client.admin.37130.log.gz 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.30655.log 2024-01-11T11:12:31.911 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29926.log.gz 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34757.log 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50703.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50703.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43877.log 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30655.log: /var/log/ceph/ceph-client.admin.34757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50783.log 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34757.log.gz 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30655.log.gz 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43877.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43877.log.gz 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29066.log 2024-01-11T11:12:31.912 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31494.log 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50783.log: /var/log/ceph/ceph-client.admin.29066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31585.log 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50783.log.gz 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29066.log.gz 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36485.log 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31494.log.gz 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45443.log 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31585.log.gz 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33025.log 2024-01-11T11:12:31.913 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36485.log.gz 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34946.log 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45443.log: /var/log/ceph/ceph-client.admin.33025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45443.log.gz 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33025.log.gz 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53382.log 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34946.log.gz 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42419.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.30242.log 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53382.log: gzip -5 --verbose -- 8.8% /var/log/ceph/ceph-client.admin.30654.log -- replaced with /var/log/ceph/ceph-client.admin.53382.log.gz 2024-01-11T11:12:31.914 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30242.log: -- /var/log/ceph/ceph-client.admin.34175.log 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30242.log.gz 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34448.log/var/log/ceph/ceph-client.admin.30654.log: 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0%/var/log/ceph/ceph-client.admin.34175.log: -- replaced with /var/log/ceph/ceph-client.admin.30654.log.gz 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.34175.log.gz 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44667.log 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34448.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34448.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47037.log 2024-01-11T11:12:31.915 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47664.log: /var/log/ceph/ceph-client.admin.44667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31396.log 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44667.log.gz 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47037.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47037.log.gz 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39931.log 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31396.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31396.log.gz 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34897.log 2024-01-11T11:12:31.916 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.39931.log: -- /var/log/ceph/ceph-osd.0.log 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39931.log.gz 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34897.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.34897.log.gz 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44102.log 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35958.log 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44102.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40721.log 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44102.log.gz 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40721.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35958.log.gz 2024-01-11T11:12:31.917 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40721.log.gz 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33114.log 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46312.log 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42738.log 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46312.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33114.log.gz 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46312.log.gz 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40802.log 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35155.log 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42738.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42738.log.gz 2024-01-11T11:12:31.918 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40802.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.40802.log.gz 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29800.log 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41282.log 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35155.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.35155.log.gz 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29800.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29800.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37535.log 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30154.log 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41282.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41282.log.gz 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37535.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49427.log 2024-01-11T11:12:31.919 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37535.log.gz 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53077.log 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30154.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30154.log.gz 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49514.log 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49427.log.gz 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48526.log 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49514.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53077.log.gz 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49514.log.gz 2024-01-11T11:12:31.920 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36525.log 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51171.log 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48526.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48526.log.gz 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36525.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36525.log.gz 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52392.log 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41594.log 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51171.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51171.log.gz 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52392.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39483.log 2024-01-11T11:12:31.921 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52392.log.gz 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47173.log 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39483.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39483.log.gz 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41594.log.gz 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37309.log 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47173.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47173.log.gz 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41416.log 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37309.log: /var/log/ceph/ceph-client.admin.40458.log: 50.8% 50.8% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37309.log.gz 2024-01-11T11:12:31.922 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33857.log 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49996.log 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41416.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41416.log.gz 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33857.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33857.log.gz 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34489.log 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49996.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49996.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36970.log 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34489.log: /var/log/ceph/ceph-client.admin.41950.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34489.log.gz 2024-01-11T11:12:31.923 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49692.log 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43280.log 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36970.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36970.log.gz 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49692.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49692.log.gz 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29590.log 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29590.log: /var/log/ceph/ceph-client.admin.43280.log: 50.4% 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29590.log.gz 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43280.log.gz 2024-01-11T11:12:31.924 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43924.log 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28960.log 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43924.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43924.log.gz 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32380.log 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.28960.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28960.log.gz 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34666.log 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32380.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32380.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38100.log 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36752.log 2024-01-11T11:12:31.925 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34666.log: /var/log/ceph/ceph-client.admin.38100.log: 50.0% 51.2% -- replaced with /var/log/ceph/ceph-client.admin.34666.log.gz 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38100.log.gz 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42691.log 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53964.log 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.36752.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36752.log.gz 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42691.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42691.log.gz 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53047.log 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31167.log 2024-01-11T11:12:31.926 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53047.log: /var/log/ceph/ceph-client.admin.53964.log: 8.8% 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53964.log.gz 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53047.log.gz 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52651.log 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35482.log 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31167.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31167.log.gz 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52651.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52651.log.gz 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51044.log 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43835.log 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51044.log: /var/log/ceph/ceph-client.admin.35482.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51044.log.gz 2024-01-11T11:12:31.927 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35482.log.gz 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43701.log 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34995.log 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43835.log.gz 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.43701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36000.log 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43701.log.gz 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.34995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30911.log 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8%/var/log/ceph/ceph-client.admin.36000.log: -- replaced with /var/log/ceph/ceph-client.admin.34995.log.gz 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.36000.log.gz 2024-01-11T11:12:31.928 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48660.log 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50870.log 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30911.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30911.log.gz 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48660.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36921.log 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48660.log.gz 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50870.log: /var/log/ceph/ceph-client.admin.36921.log: 50.0% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.50870.log.gz 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36921.log.gz 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40937.log 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51862.log 2024-01-11T11:12:31.929 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44757.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40937.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40937.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41860.log 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40329.log 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51862.log: /var/log/ceph/ceph-client.admin.41860.log: 50.0% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.51862.log.gz 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41860.log.gz 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49093.log 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41322.log 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40329.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.40329.log.gz 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49093.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49093.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42034.log 2024-01-11T11:12:31.930 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31912.log 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.41322.log: /var/log/ceph/ceph-client.admin.42034.log: 50.4% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41322.log.gz 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42034.log.gz 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31892.log 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29422.log 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31912.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31912.log.gz 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31892.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31892.log.gz 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52608.log 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50433.log 2024-01-11T11:12:31.931 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52608.log: /var/log/ceph/ceph-client.admin.29422.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.52608.log.gz 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29422.log.gz 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29108.log 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53473.log 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50433.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50433.log.gz 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29108.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29108.log.gz 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46863.log 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38589.log 2024-01-11T11:12:31.932 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46863.log: /var/log/ceph/ceph-client.admin.53473.log: 8.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.53473.log.gz 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46863.log.gz 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42916.log 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35393.log 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38589.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38589.log.gz 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42916.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43363.log 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42916.log.gz 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37090.log 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.35393.log: /var/log/ceph/ceph-client.admin.43363.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43363.log.gz 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.35393.log.gz 2024-01-11T11:12:31.933 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31714.log 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.37090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52861.log 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.37090.log.gz 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52861.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31714.log.gz 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52861.log.gz 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48880.log 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38827.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.42644.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43788.log 2024-01-11T11:12:31.934 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53580.log 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48880.log: /var/log/ceph/ceph-client.admin.43788.log: 49.6% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.43788.log.gz 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48880.log.gz 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44413.log 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49650.log 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53580.log.gz 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48700.log 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44413.log.gz 2024-01-11T11:12:31.935 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39791.log 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0%/var/log/ceph/ceph-client.admin.48700.log: -- replaced with /var/log/ceph/ceph-client.admin.49650.log.gz 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48700.log.gz 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33421.log 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32102.log 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39791.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39791.log.gz 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33421.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48566.log 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33421.log.gz 2024-01-11T11:12:31.936 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29293.log 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32102.log: /var/log/ceph/ceph-client.admin.48566.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32102.log.gz 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48566.log.gz 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52040.log 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45490.log 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.29293.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29293.log.gz 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.52040.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52040.log.gz 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30113.log 2024-01-11T11:12:31.937 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31047.log 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30113.log: /var/log/ceph/ceph-client.admin.45490.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30113.log.gz 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45490.log.gz 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39176.log 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40761.log 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.31047.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31047.log.gz 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39176.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39176.log.gz 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53537.log 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40761.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40761.log.gz 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30998.log 2024-01-11T11:12:31.938 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53537.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53537.log.gz 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30556.log 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37655.log 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30998.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30998.log.gz 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30556.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30556.log.gz 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38658.log 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38658.log: /var/log/ceph/ceph-client.admin.37655.log: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.38658.log.gz 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37655.log.gz 2024-01-11T11:12:31.939 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38420.log 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.45626.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50937.log 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.38420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32022.log 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50937.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.38420.log.gz 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50937.log.gz 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45579.log 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51124.log 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32022.log: /var/log/ceph/ceph-client.admin.45579.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32022.log.gz 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45579.log.gz 2024-01-11T11:12:31.940 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32905.log 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49823.log 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51124.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.51124.log.gz 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32905.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32905.log.gz 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44457.log 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50663.log 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44457.log: /var/log/ceph/ceph-client.admin.49823.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.44457.log.gz 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.49823.log.gz 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47220.log 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45987.log 2024-01-11T11:12:31.941 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50663.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50663.log.gz 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47220.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47220.log.gz 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46291.log 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49467.log 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.46291.log: /var/log/ceph/ceph-client.admin.45987.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46291.log.gz 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45987.log.gz 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39096.log 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29758.log 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49467.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49467.log.gz 2024-01-11T11:12:31.942 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39096.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.39096.log.gz 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48922.log 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33163.log 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48922.log: /var/log/ceph/ceph-client.admin.29758.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48922.log.gz 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29758.log.gz 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49561.log 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33163.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53200.log 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33163.log.gz 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.49561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53290.log 2024-01-11T11:12:31.943 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53200.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49561.log.gz 8.0% -- replaced with /var/log/ceph/ceph-client.admin.53200.log.gz 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.53290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30869.log 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.53290.log.gz 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.44799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36216.log 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.30869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32807.log 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30869.log.gz/var/log/ceph/ceph-client.admin.36216.log: 2024-01-11T11:12:31.944 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.36216.log.gz 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51993.log 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33470.log 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32807.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32807.log.gz 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.51993.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42094.log 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51993.log.gz 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39699.log 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33470.log: /var/log/ceph/ceph-client.admin.42094.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33470.log.gz 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42094.log.gz 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:31.945 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40979.log 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.39699.log: 50.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32291.log 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39699.log.gz 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48749.log 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.40979.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40979.log.gz 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.32291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50823.log 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32291.log.gz 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.48749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33986.log 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.50823.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48749.log.gz 2024-01-11T11:12:31.946 INFO:teuthology.orchestra.run.smithi040.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50823.log.gz 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48406.log 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47262.log 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.33986.log: /var/log/ceph/ceph-client.admin.48406.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.33986.log.gz 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.48406.log.gz 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34806.log 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-client.admin.47262.log: /var/log/ceph/ceph-client.admin.34806.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47262.log.gz 2024-01-11T11:12:31.947 INFO:teuthology.orchestra.run.smithi040.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34806.log.gz 2024-01-11T11:12:31.948 INFO:teuthology.orchestra.run.smithi040.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-11T11:12:32.432 INFO:teuthology.orchestra.run.smithi040.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.0.39286.log.gz 2024-01-11T11:12:32.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3307 16 374 358 0.00676515 0 - 0.0803724 2024-01-11T11:12:32.607 INFO:teuthology.orchestra.run.smithi040.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.0.30941.log.gz 2024-01-11T11:12:32.619 INFO:teuthology.orchestra.run.smithi040.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.0.44547.log.gz 2024-01-11T11:12:33.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3308 16 374 358 0.0067631 0 - 0.0803724 2024-01-11T11:12:34.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3309 16 374 358 0.00676106 0 - 0.0803724 2024-01-11T11:12:35.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3310 16 374 358 0.00675902 0 - 0.0803724 2024-01-11T11:12:36.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3311 16 374 358 0.00675698 0 - 0.0803724 2024-01-11T11:12:37.523 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3312 16 374 358 0.00675494 0 - 0.0803724 2024-01-11T11:12:38.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3313 16 374 358 0.0067529 0 - 0.0803724 2024-01-11T11:12:38.666 INFO:teuthology.orchestra.run.smithi134.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-11T11:12:39.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3314 16 374 358 0.00675086 0 - 0.0803724 2024-01-11T11:12:40.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3315 16 374 358 0.00674882 0 - 0.0803724 2024-01-11T11:12:41.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3316 16 374 358 0.00674679 0 - 0.0803724 2024-01-11T11:12:42.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3317 16 374 358 0.00674475 0 - 0.0803724 2024-01-11T11:12:43.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3318 16 374 358 0.00674272 0 - 0.0803724 2024-01-11T11:12:44.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3319 16 374 358 0.00674069 0 - 0.0803724 2024-01-11T11:12:45.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:12:45.522278+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:12:45.525 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:12:45.525 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3320 16 374 358 0.00673866 0 - 0.0803724 2024-01-11T11:12:46.525 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3321 16 374 358 0.00673663 0 - 0.0803724 2024-01-11T11:12:47.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3322 16 374 358 0.0067346 0 - 0.0803724 2024-01-11T11:12:48.355 INFO:teuthology.orchestra.run.smithi040.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-11T11:12:48.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3323 16 374 358 0.00673258 0 - 0.0803724 2024-01-11T11:12:49.405 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-11T11:12:49.406 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-01-11T11:12:49.406 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m17.844s 2024-01-11T11:12:49.406 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m24.042s 2024-01-11T11:12:49.407 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.921s 2024-01-11T11:12:49.524 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3324 16 374 358 0.00673055 0 - 0.0803724 2024-01-11T11:12:50.525 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3325 16 374 358 0.00672853 0 - 0.0803724 2024-01-11T11:12:51.525 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3326 16 374 358 0.0067265 0 - 0.0803724 2024-01-11T11:12:51.812 INFO:teuthology.orchestra.run.smithi040.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-11T11:12:51.814 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-01-11T11:12:51.814 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m20.292s 2024-01-11T11:12:51.814 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m39.065s 2024-01-11T11:12:51.814 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m1.548s 2024-01-11T11:12:51.815 INFO:tasks.ceph:Archiving logs... 2024-01-11T11:12:51.815 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/remote/smithi040/log 2024-01-11T11:12:51.816 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-11T11:12:52.539 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3327 16 374 358 0.00672448 0 - 0.0803724 2024-01-11T11:12:53.553 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3328 16 374 358 0.00672246 0 - 0.0803724 2024-01-11T11:12:54.642 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3329 16 374 358 0.00672044 0 - 0.0803724 2024-01-11T11:12:55.556 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3330 16 374 358 0.00671842 0 - 0.0803724 2024-01-11T11:12:56.543 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3331 16 374 358 0.00671641 0 - 0.0803724 2024-01-11T11:12:57.540 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3332 16 374 358 0.00671439 0 - 0.0803724 2024-01-11T11:12:58.293 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/remote/smithi134/log 2024-01-11T11:12:58.294 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-11T11:12:58.531 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3333 16 374 358 0.00671238 0 - 0.0803724 2024-01-11T11:12:59.542 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3334 16 374 358 0.00671036 0 - 0.0803724 2024-01-11T11:13:00.528 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3335 16 374 358 0.00670835 0 - 0.0803724 2024-01-11T11:13:01.533 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3336 16 374 358 0.00670634 0 - 0.0803724 2024-01-11T11:13:02.379 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2774, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2606, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2372, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-01-11T11:13:02.397 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-11T11:13:02.406 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/thrashosds.py", line 215, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2808, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2798, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2561, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 2553, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1627, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph_f14f60afd567d61490bfe92caba04c0ffb5b09d2/qa/tasks/ceph_manager.py", line 1618, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-01-11T11:13:02.421 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-11T11:13:02.421 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-11T11:13:02.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-11T11:13:02.500 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-11T11:13:02.510 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-11T11:13:02.511 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T11:13:02.513 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-11T11:13:02.526 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3337 16 374 358 0.00670433 0 - 0.0803724 2024-01-11T11:13:02.527 INFO:teuthology.orchestra.run.smithi040.stderr:bash: line 1: ntpq: command not found 2024-01-11T11:13:02.529 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-01-11T11:13:02.531 INFO:teuthology.orchestra.run.smithi040.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T11:13:02.531 INFO:teuthology.orchestra.run.smithi040.stdout:=============================================================================== 2024-01-11T11:13:02.531 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 59 -680us[ -711us] +/- 74ms 2024-01-11T11:13:02.531 INFO:teuthology.orchestra.run.smithi040.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 +1040us[+1009us] +/- 68ms 2024-01-11T11:13:02.531 INFO:teuthology.orchestra.run.smithi040.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 -453us[ -484us] +/- 73ms 2024-01-11T11:13:02.531 INFO:teuthology.orchestra.run.smithi040.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T11:13:02.534 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-11T11:13:02.534 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-01-11T11:13:02.534 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 121 -717us[ -750us] +/- 73ms 2024-01-11T11:13:02.534 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 +1012us[+1012us] +/- 68ms 2024-01-11T11:13:02.534 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 56 -470us[ -471us] +/- 73ms 2024-01-11T11:13:02.534 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-01-11T11:13:02.535 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-11T11:13:02.545 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-11T11:13:02.545 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-11T11:13:02.554 DEBUG:teuthology.orchestra.run.smithi040:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-11T11:13:02.653 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-11T11:13:02.721 DEBUG:teuthology.orchestra.run.smithi040:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T11:13:02.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T11:13:02.752 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-11T11:13:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T11:13:02.784 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-11T11:13:02.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-11T11:13:02.806 INFO:teuthology.task.internal:Duration was 4400.777191 seconds 2024-01-11T11:13:02.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-11T11:13:02.820 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-11T11:13:02.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-11T11:13:02.823 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-11T11:13:02.864 INFO:teuthology.orchestra.run.smithi040.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T11:13:02.869 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-11T11:13:03.250 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-11T11:13:03.250 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-01-11T11:13:03.250 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-11T11:13:03.309 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-01-11T11:13:03.309 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-11T11:13:03.337 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-11T11:13:03.337 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-11T11:13:03.351 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-11T11:13:03.406 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-11T11:13:03.407 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-11T11:13:03.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-11T11:13:03.526 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3338 16 374 358 0.00670232 0 - 0.0803724 2024-01-11T11:13:03.542 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-11T11:13:03.552 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-11T11:13:03.552 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-11T11:13:03.579 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-11T11:13:03.606 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-11T11:13:03.617 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-11T11:13:03.622 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-11T11:13:03.645 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-01-11T11:13:03.672 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-01-11T11:13:03.692 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-11T11:13:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T11:13:03.742 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-11T11:13:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-11T11:13:03.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-11T11:13:03.766 INFO:teuthology.task.internal:Transferring archived files... 2024-01-11T11:13:03.767 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/remote/smithi040 2024-01-11T11:13:03.767 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-11T11:13:03.817 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429/remote/smithi134 2024-01-11T11:13:03.817 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-11T11:13:03.851 INFO:teuthology.task.internal:Removing archive directory... 2024-01-11T11:13:03.852 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-11T11:13:03.854 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-11T11:13:03.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-11T11:13:03.912 INFO:teuthology.task.internal:Not uploading archives. 2024-01-11T11:13:03.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-11T11:13:03.922 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-11T11:13:03.922 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-11T11:13:03.925 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-11T11:13:03.937 INFO:teuthology.orchestra.run.smithi040.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 11 11:13 /home/ubuntu/cephtest 2024-01-11T11:13:03.955 INFO:teuthology.orchestra.run.smithi134.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 11 11:13 /home/ubuntu/cephtest 2024-01-11T11:13:03.956 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-11T11:13:03.967 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-11T11:13:03.979 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi040 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2024-01-11T11:13:03.980 INFO:teuthology.nuke:Checking targets against current locks 2024-01-11T11:13:04.006 DEBUG:teuthology.nuke:shortname: smithi040 2024-01-11T11:13:04.007 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T11:13:04.031 DEBUG:teuthology.nuke:shortname: smithi134 2024-01-11T11:13:04.032 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-11T11:13:04.035 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 09:50:36.206849', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP9FCjzta3j/kTl2AOjmnxEItzktKfjKINZVSKfOMgwX2Ba7WsHwXv+m5Q1BXECZwcz6kkuR8ASgaWKtlP9UxeI='} 2024-01-11T11:13:04.057 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-11_09:46:59-crimson-rados-wip-matanb-crimson-multicore-msgr-distro-crimson-smithi/7513429', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-11 09:50:36.208506', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBEp1dt+7AFCwJg78Y81xcft8Bn5sEh6vOHvj2vbxh5jm9Qm1g1pyyMHmf1Suc/91Z1i3t0pR8S2AcXCmiT4zS4='} 2024-01-11T11:13:04.106 INFO:teuthology.orchestra.console.smithi134:Power off 2024-01-11T11:13:04.107 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-11T11:13:04.124 INFO:teuthology.orchestra.console.smithi040:Power off 2024-01-11T11:13:04.125 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-11T11:13:04.134 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-01-11T11:13:04.134 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:04.148 DEBUG:teuthology.orchestra.console.smithi040:power off output: Chassis Power Control: Down/Off 2024-01-11T11:13:04.148 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:04.158 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-11T11:13:04.173 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-01-11T11:13:04.527 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3339 16 374 358 0.00670031 0 - 0.0803724 2024-01-11T11:13:05.527 INFO:tasks.radosbench.radosbench.0.smithi040.stdout:2024-01-11T11:13:05.524772+0000 min lat: 0.00968733 max lat: 0.251984 avg lat: 0.0803724 2024-01-11T11:13:05.527 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-01-11T11:13:05.527 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3340 16 374 358 0.00669831 0 - 0.0803724 2024-01-11T11:13:06.527 INFO:tasks.radosbench.radosbench.0.smithi040.stdout: 3341 16 374 358 0.0066963 0 - 0.0803724 2024-01-11T11:13:08.160 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:08.174 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:08.271 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-11T11:13:08.284 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is on 2024-01-11T11:13:12.276 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:12.285 DEBUG:teuthology.orchestra.console.smithi040:pexpect command: ipmitool -H smithi040.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:12.387 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-01-11T11:13:12.394 DEBUG:teuthology.orchestra.console.smithi040:check power output: Chassis Power is off 2024-01-11T11:13:12.495 INFO:teuthology.orchestra.console.smithi040:Power off completed 2024-01-11T11:13:16.392 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-11T11:13:16.503 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-01-11T11:13:16.604 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-01-11T11:13:16.708 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} duration: 4400.777190923691 failure_reason: reached maximum tries (501) after waiting for 3000 seconds flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0832196059b14ea697e5435449452a61 status: fail success: false 2024-01-11T11:13:16.709 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-11T11:13:16.797 INFO:teuthology.run:FAIL